Face identity privacy protection method and system based on makeup style migration

Abstract

The invention belongs to an image processing technology, and particularly relates to a face identity privacy protection method and system based on makeup style migration. The method comprises the following steps: training a conditional diffusion model by using a face image and corresponding identity features thereof, so that the conditional diffusion model has the capability of generating an image with specific identity features; inputting the identity features of the original image and the target image into a pre-trained diffusion model, and guiding a sampling process through multi-modal information (such as identity semantics, structural constraint, makeup migration and the like) to generate a new image which is highly similar to the original image visually but cannot be recognized by a machine; according to the invention, it is ensured that the processed image is visually similar to the original image, and it is ensured that the processed image effectively destroys identity information which can be identified by a machine, so that image functionality and privacy security are both considered in a privacy protection scene.

Description

Technical Field

[0001] This invention pertains to image processing technology, and specifically relates to a method and system for protecting facial identity privacy based on makeup style transfer. Background Technology

[0002] With breakthroughs in deep learning technology in computer vision, the accuracy and performance of facial recognition systems have been significantly improved, making it a key technology supporting various entertainment activities and commercial services. While this technology facilitates daily user interactions (such as identity verification and personalized recommendations) by analyzing biometric features in facial images or videos, it also raises serious privacy and security concerns. User images and video content on social media platforms may be efficiently and covertly collected and analyzed by unauthorized commercial services using open-source facial recognition models, and such data collection often lacks necessary security restrictions. Because facial images, as highly sensitive biometric data, contain rich identity information, their large-scale, uncontrolled circulation directly threatens individual privacy rights. Although the current legal framework for personal information protection is gradually improving, from a technical perspective, there is still an urgent need for a solution that can effectively disrupt machine-identifiable identity information (such as the detectability of embedded biometric features) while ensuring the visual usability of facial images (such as maintaining natural appearance and recognizability).

[0003] In the research field, existing methods for protecting facial image identity privacy can be summarized into two main categories:

[0004] 1) Adversarial Face Privacy Protection Methods Based on Noise Perturbation. Early methods in this category relied heavily on traditional image obfuscation techniques (such as pixelation, blurring, or mosaic masking) to reduce image sharpness and hinder feature extraction by face recognition systems. However, these methods often significantly sacrifice the visual usability of the image (e.g., causing blurred facial details and distorted appearance). Subsequent improvements shifted to injecting subtle, imperceptible perturbations (i.e., adversarial perturbations) into the original image. These perturbation model features prevent unauthorized face recognition systems from accurately determining identity and other attributes. However, when generating adversarial perturbations, these methods often leave weak noise traces in localized areas of the image, which may still be noticeable upon close inspection, thus affecting the natural appearance and practical application value of the image.

[0005] 2) Adversarial Face Privacy Protection Methods Based on Image Generation. These methods utilize generative models (such as GANs and diffusion models) to reconstruct face images. During the generation process, conditional guidance is used to ensure the output image maintains a high degree of visual consistency with the original image, while simultaneously disrupting the machine's ability to correctly identify the face. For example, the CLIP2Protect method proposed by Shamshad et al. employs a two-step strategy: combining a generative model with user-defined text prompts (such as "makeup effect") to generate an image that possesses both aesthetic enhancement characteristics and the ability to resist face recognition systems. However, existing generative methods generally suffer from insufficient control over adversarial perturbations to identity information during the generation process—perturbations introduced to disrupt machine recognition capabilities (such as feature decoupling bias, generation artifacts, etc.) can easily lead to a decline in the quality of the processed image (such as color distortion, texture abnormalities, or structural deformation), making it difficult to balance visual usability with the strength of privacy protection. Summary of the Invention

[0006] To address the problems existing in current mainstream facial image identity privacy protection technologies, this invention proposes a facial identity privacy protection method and system based on makeup style transfer. The method first inputs the identity features of the original image and the target image into a pre-trained conditional diffusion model, and combines a sampling strategy guided by multimodal information to achieve the privacy protection effect of effectively destroying the machine-recognizable identity information while preserving the visual usability of the facial image.

[0007] Furthermore, a conditional diffusion model based on identity features is selected to generate identity-anonymous images. This conditional diffusion model includes an identity feature encoding module, a noise prediction module, and an image decoding and generation module. During the model training phase, identity features and image features are deeply fused during the diffusion process. Images with specific identity representation are generated through the sampling process. Finally, through multimodal guidance, the coordinated control of identity replacement, makeup transfer, and structure preservation is achieved to generate identity-anonymous images.

[0008] Furthermore, in order to enable the conditional diffusion model based on facial identity features to generate images with specific identity features, a loss function is used to update the network parameters. The loss function used includes at least the identity information loss between the generated image identity features and the input identity features, as well as the noise loss between the diffusion model's predicted noise and the actual noise.

[0009] Furthermore, the loss function used is expressed as:

[0010]

[0011] in, This represents the total loss function used to train the conditional diffusion model; To determine the loss of identity information between the generated image identity features and the input identity features; The noise loss between the predicted noise and the actual noise is determined by the diffusion model; λ is... The weight.

[0012] Furthermore, the loss of identity information between the generated image identity features and the input target identity features. Represented as:

[0013]

[0014] Among them, V id This represents the target identity features input to the diffusion model. This represents the identity feature extracted from the generated image by the face recognition model. Both are feature vectors extracted by the pre-trained face recognition model, and cos(.) represents the cosine similarity.

[0015] Furthermore, the diffusion model compares the predicted noise with the actual noise level. Represented as:

[0016]

[0017] Where t is the time step of the diffusion process, X t V is the noisy image of the original image at step t. id Let ∈ be the target identity feature input to the diffusion model, and ∈ be the actual noise added to the original image. θ (.) represents noise in the model prediction.

[0018] Furthermore, based on the conditional diffusion model trained with identity features as the condition, the sampling process is guided by multimodal information to generate anonymous images that retain the target identity, reference makeup, and original image structure. The multimodal information used includes at least: target identity information guiding the generated image to resemble the target image, makeup information guiding the generated image to resemble the reference image, and facial structure information guiding the generated image to maintain consistency with the original image.

[0019] Furthermore, multimodal information guides the sampling process by updating the gradient using a loss function, which is:

[0020]

[0021] in, This represents the total loss function for multimodal guidance. This represents the adversarial loss of identity information between the generated image and the target image; This represents the loss of facial structural information between the generated image and the original image; λ represents the loss of makeup style information between the generated image and the makeup reference image; adv , λ sem , λ make They are respectively The weight.

[0022] Furthermore, identity information guidance employs an identity privacy enhancement strategy: K pre-trained face recognition models with high recognition accuracy are selected as proxy models to jointly guide identity information, aiming to explore a general adversarial makeup domain and improve the anonymity of generated images under different recognition models. Identity information guidance loss. Represented as:

[0023]

[0024] Among them, F k Let X represent the k-th pre-trained face recognition model. p X represents the generated anonymous image. tar This represents the target image.

[0025] Among them, F k Let X represent the k-th pre-trained face recognition model. p X represents the generated anonymous image. tar The target image is represented by this loss. By minimizing the identity similarity between the generated image and the target image under multiple recognition models, identity confusion and adversarial anonymity are achieved, making it difficult for the generated image to be recognized as the original identity under multiple recognizers.

[0026] This invention also proposes a facial identity privacy protection system based on makeup style transfer, including a conditional diffusion model with identity features as conditions. The conditional diffusion model generates an image with input identity features by using the input original image and identity features. During the sampling stage of the diffusion model, the multimodal information guidance module guides the noisy image with multi-source information to generate an anonymous image with the facial structure of the original image, the makeup style of the reference image, and the identity features of the target image.

[0027] This invention, through a pre-trained conditional diffusion model based on identity features and a diffusion sampling strategy guided by multimodal information, ensures that the generated images remain natural and usable to the human eye, while providing strong anonymity for machine recognition models. This effectively resolves the contradiction between privacy protection and visual usability in facial identity privacy images. Specific beneficial effects of this invention include:

[0028] 1) The present invention has strong usability, that is, in terms of visual diversity, the present invention can generate facial identity privacy protection images with different makeup styles by changing the reference images of different makeup styles, so as to meet the personalized needs of different scenarios.

[0029] 2) This invention has high efficiency. After training, the conditional diffusion model based on identity features proposed in this method only needs to adjust the multimodal guidance information (such as target identity, makeup image, etc.) without retraining the model, which can quickly generate diverse anonymous images, significantly improving generation efficiency and flexibility.

[0030] 3) This invention has strong generalization and robustness. By adopting a multi-model joint guidance strategy, it improves the anonymity of the generated image for different face recognition algorithms and has stronger adaptability to practical applications. Attached Figure Description

[0031] Figure 1 This is a flowchart of a facial identity privacy protection method based on makeup style transfer according to the present invention;

[0032] Figure 2 This is a schematic diagram of the conditional diffusion model training method in this invention;

[0033] Figure 3 This is a schematic diagram of the multimodal guidance method in this invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] This invention proposes a facial identity privacy protection method based on makeup style transfer. By training a conditional diffusion model with facial identity features as the condition, the identity features of the original image and the target image are input into the pre-trained conditional diffusion model. Combined with a sampling strategy guided by multimodal information, an anonymous image is generated. While preserving the visual usability of the image, it effectively destroys its machine-recognizable identity information.

[0036] Example

[0037] In this embodiment, as Figure 1 This invention is a method for protecting facial identity privacy based on makeup style transfer, specifically including the following steps:

[0038] By using face images and their corresponding face identity features, a conditional diffusion model conditioned on identity features is trained, enabling the model to generate images with corresponding identity representation based on input identity features.

[0039] After the model training is completed, the identity features of the original image and the target image are input into the pre-trained conditional diffusion model. During the sampling process of the diffusion model, a multimodal information guidance mechanism is introduced, including but not limited to identity semantic guidance, structural constraint guidance and makeup transfer guidance, to control the generation process in multiple dimensions and finally output an anonymous image.

[0040] Anonymous images are visually consistent with or highly similar to the original images, meeting the needs of daily visual recognition and image use; at the machine recognition level, they cannot be correctly identified as the original identity by facial recognition models, effectively protecting the privacy of identity information.

[0041] The conditional diffusion model used in this embodiment, which is a generative model conditioned on facial identity features, typically includes a noise prediction network, a feature encoding module, and a decoding and generation module. During training, it is constrained by two loss functions: an identity loss function and a noise loss function. The identity loss function ensures that the identity features of the generated image are consistent with the target identity features in the feature space, thus ensuring that the generated image possesses the visual and recognizable features of the target identity. The noise loss function ensures that the diffusion model can accurately predict noise during the denoising and generation process, thereby improving the overall quality and clarity of the generated image. The total loss function includes:

[0042]

[0043] in, This represents the total loss function used to train the conditional diffusion model; This represents the loss of identity information between the generated image identity features and the input identity features; This represents the noise loss between the noise predicted by the diffusion model and the actual noise; λ is... The weights. The identity information loss between the generated image identity features and the input identity features. Represented as: X t-1

[0044]

[0045] Where cos(.) is the cosine distance between the generated image identity features and the input identity features, V id The target identity features are used as input in the diffusion model. To generate the identity feature vector extracted from the image using a face recognition model. V id All of these are facial image identity features extracted by the feature extractor of a pre-trained facial recognition model.

[0046] In this embodiment, the noise loss between the diffusion model's predicted noise and the actual noise is compared. Represented as:

[0047]

[0048] Where t represents the t-th time step of the diffusion process, X t V is the image obtained after t steps of noise addition processing on the original face image. id Here, ∈ represents the identity features input to the model, and ∈ represents the noise actually added to the image. θ (.) represents the noise predicted by the diffusion model.

[0049] Building upon the pre-trained conditional diffusion model, a multimodal information-guided sampling strategy is further employed. This strategy introduces various guiding information during the sampling phase of the diffusion model, enabling multi-dimensional control over the generation process to generate identity-anonymized images that meet specific requirements. The multimodal information used includes, but is not limited to, identity information from the target image to guide the generation of an anonymous image, makeup information from a reference makeup image to ensure consistency in makeup style, and facial structure information from the original image to maintain the original facial structure and avoid structural distortion. This multimodal information guidance influences the sampling process by updating the gradient using a loss function. Represented as:

[0050]

[0051] in, This represents the total loss function for multimodal guidance. This represents the loss of identity information between the identity features of the generated image and the identity features of the target image. This represents the loss of facial structural information between the generated image and the original image; This represents the loss of makeup style information between the generated image and the makeup reference image; They are respectively The weights are determined by identity information guidance. An identity privacy enhancement strategy is introduced, which selects K pre-trained face recognition models with high accuracy in practical applications as proxy models to jointly constrain the identity features of the generated image. This aims to uncover universally adversarial identity obfuscation regions, improving the anonymity of the generated image across multiple recognition models. The identity information guidance loss... Represented as:

[0052]

[0053] Among them, F k (.) represents the feature vector extracted by the k-th pre-trained face recognition model, X p To generate an anonymous image, X tar Image for target identity.

[0054] This invention also proposes a facial identity privacy protection system based on makeup style transfer, including a conditional diffusion model with identity features as conditions and a multimodal information guidance module. The conditional diffusion model generates an image with input identity features by using the input original image and identity features. During the sampling stage of the diffusion model, the multimodal information guidance module guides the noisy image to generate an anonymous image with the facial structure of the original image, the makeup style of the reference image, and the identity features of the target image.

[0055] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A face identity privacy protection method based on makeup style transfer, characterized in that, The original face image and the target image identity feature are input into the pre-trained conditional diffusion model, and the multi-modal information is used to guide the model sampling process to generate an identity anonymous image which effectively destroys the machine recognizable face identity information while retaining the visual usability.

2. The method of claim 1, wherein, Before generating the identity anonymous image, the conditional diffusion model is trained using face images and their corresponding identity features, so that the model can generate images with corresponding identity features according to the input identity features.

3. The method of claim 2, wherein, In order to make the conditional diffusion model have the ability to generate images with specific identity features, a loss function is used to update the network parameters, and the loss function includes at least the identity information loss between the generated image identity features and the input identity features, and the noise loss between the diffusion model predicted noise and the actual noise.

4. The method of claim 3, wherein, The loss function is represented as: wherein, represents a total loss function of training the conditional diffusion model; represents an identity information loss between the generated image identity feature and the input identity feature; represents a noise loss between the diffusion model predicted noise and the actual noise; λ is a weight of .

5. The method of claim 4, wherein, generating an identity information loss between the image identity feature and the input identity feature is represented as: wherein, denotes the cosine distance between the identity feature of the generated image and the input identity feature, V id denotes the identity feature in the input diffusion model, denotes the identity feature of the generated image extracted by the pre-trained face recognition model, V id are face image identity feature vectors extracted by the feature extractor of the pre-trained face recognition model.

6. The method of claim 4, wherein, The diffusion model predicts a noise loss between the noise and the actual noise is represented as: where t represents the time step in the diffusion model adding noise process, X t represents the image obtained after t-step noise addition to the original face image, V id is the input identity feature, ∈ is the noise actually added to the original image ∈ θ (.) is the noise predicted by the diffusion model.

7. The method of claim 1, wherein, The multi-modal information guided sampling process includes at least the following guiding information: guiding the identity feature of the generated image to tend towards the target identity information of the target image, guiding the makeup style of the generated image to tend towards the makeup information of the reference image, and guiding the face structure of the generated image to remain consistent with the original image, thereby generating an identity anonymous image with target identity, reference makeup and original image structure features.

8. The method of claim 7, wherein, The multi-modal information guidance affects the conditional diffusion model sampling process in the form of loss update gradient, and the loss function used is: wherein, represents the total loss function of multi-modal information guidance; represents the identity information loss between the generated image identity feature and the target image identity feature; represents the face structure information loss between the generated image and the original image; represents the makeup style information loss between the generated image and the makeup reference image; λ adv , λ sem , λ make respectively, the weight of.

9. The method of claim 8, wherein, The identity information guidance adopts an identity privacy enhancement strategy, including selecting K pre-trained face recognition models with high recognition accuracy as proxy models to jointly constrain the identity features of the generated image to excavate a general adversarial makeup domain, and identity information guidance loss is represented as: wherein F k represents the kth pre-trained face recognition model, X p represents the generated identity-anonymized image, X tar represents the target identity image.

10. A face identity privacy protection system based on makeup style transfer, characterized in that, The conditional diffusion model includes an identity feature conditional diffusion model, a multi-modal information guiding module, and the conditional diffusion model generates images with corresponding identity features according to the input original face image and identity feature. The multi-modal information guiding module guides the noise image during the sampling stage of the conditional diffusion model to generate an identity anonymous image with the original image face structure, the reference image makeup style, and the target image identity feature.

Images