Face image privacy protection method based on collaborative diffusion model
Through the face image privacy protection method based on the collaborative diffusion model, using multiple information extractors and variational autoencoders, the problem of lack of general face image privacy protection solutions in the prior art is solved, and the situation is realized in multiple scenarios High-quality privacy protection.
Patent Information
- Application Number
- CN202411089862.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-12-03
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The prior art lacks a general method of face image privacy protection and cannot be flexibly applied in multiple scenarios. The existing solutions are usually tailor-made for specific scenarios, making it difficult to cope with new needs.
Using a face image privacy protection method based on the collaborative diffusion model, by constructing a variety of information extractors and variational autoencoders, combining the conditional sub-diffusion model, dynamically dispatching each sub-model to generate faces with high quality and privacy protection capabilities. image.
It realizes multifunctional and fine-grained privacy protection while ensuring image quality. It has wide application prospects and improves the quality and robustness of generated images.
Smart Images

Figure CN119066701A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of information security technology, and specifically is a facial image privacy protection method based on a collaborative diffusion model. The present invention can efficiently generate facial images with high quality and privacy protection capabilities by constructing multiple information extractors, combining variational autoencoders and conditional sub-diffusion models. The quality and robustness of the generated images are further improved by dynamically scheduling each sub-model through the collaborative diffusion model. The present invention can achieve multifunctional and fine-grained privacy protection while ensuring image quality, and has broad application prospects. Background Art
[0002] With the development of computer vision technology, face recognition technology has made significant progress and has been widely used in many fields, such as security monitoring, identity authentication, social media, etc. However, this also brings serious privacy protection issues, especially in the collection, storage and use of face image data, there is a risk of leakage and abuse. The privacy protection of face images has become a topic of great concern. In view of this, many face privacy protection schemes have been proposed, which either focus on identity privacy protection or attribute privacy protection. These schemes are usually tailored to specific scenarios. When facing new scenarios and new requirements, it often takes a lot of time to redesign the entire process. At present, we lack a general method that can cope with multiple scenarios. In recent years, privacy protection methods based on diffusion models have gradually attracted the attention of researchers. Diffusion models can generate images similar to the original images but without sensitive information by learning the distribution of data, thereby achieving privacy protection. Due to its high-quality generation ability and unique generation method, diffusion models have the potential for flexible application in multiple scenarios. How to use diffusion models to design a general face image privacy protection technology solution urgently needs to be studied. Summary of the invention
[0003] The technical problem to be solved by the present invention is how to implement a universal technical solution for protecting the privacy of facial images.
[0004] The present invention adopts the following technical solutions to solve the above technical problems:
[0005] A face image privacy protection method based on a collaborative diffusion model comprises the following steps:
[0006] Step 1), select a face image dataset, process the face images therein into the same size of 512×512, and perform pixel normalization on the face images;
[0007] Step 2), constructing a face identity information extractor, a face structure information extractor, and an image content information extractor according to the face image identity, face structure, and image content;
[0008] Step 3), construct and train a face image generation network;
[0009] Step 3.1), construct a face image generation network based on the collaborative diffusion model;
[0010] Step 3.2), construct the loss function for face image generation;
[0011] Step 3.3), input each image in the preprocessed face image data set into the face identity information extractor, the facial structure information extractor, and the image content information extractor, and complete the training in combination with the loss function generated by the face image to obtain a trained face image generation network;
[0012] Step 4), input the image that needs to be protected for face image privacy into the trained face image generation network to obtain a protected image.
[0013] As a further optimization scheme of the face image privacy protection method based on the collaborative diffusion model of the present invention, the specific steps of step 2) are as follows:
[0014] Step 2.1), construct the face identity information extractor f1 of the Inception network;
[0015] Step 2.2), construct the facial structure information extractor f2 of the BiSeNet network;
[0016] Step 2.3), if the face image dataset is an image dataset, the network structure constructed is the image content information extractor of the VisionTransformer network for encoding the image data; if the face image dataset is a picture-text pair dataset, the network structure constructed is the image content information extractor of the Transformer network for encoding the text data; let the constructed image content information extractor be f3.
[0017] As a further optimization scheme of the face image privacy protection method based on the collaborative diffusion model of the present invention, the detailed steps of step 3.1) are as follows:
[0018] Step 3.1.1), construct a variational autoencoder model, which consists of an encoder network E and a decoder network D. The encoder network E compresses each face image in the face image dataset into a latent space representation z, and then the decoder network D reconstructs the latent space representation z into a reconstructed image;
[0019] Step 3.1.2), construct an identity conditional sub-diffusion model. The identity conditional sub-diffusion model guides the generation of faces by introducing face identity tensors in the generation process. Its network structure is conditional U-Net.
[0020] Step 3.1.3), construct a facial structure conditional diffusion model. The facial structure conditional diffusion model guides the generation of faces by introducing facial structure tensors in the generation process. Its network structure is a conditional U-Net.
[0021] Step 3.1.4), construct a text conditional sub-diffusion model. The text conditional diffusion model guides the generation of faces by introducing image content tensors in the generation process. Its network structure is conditional U-Net.
[0022] Step 3.1.5), construct a collaborative diffusion model. The collaborative diffusion model uses a dynamic scheduler to perform weighted averaging on the identity conditional sub-diffusion model, the facial structure conditional sub-diffusion model, and the text conditional sub-diffusion model to form a face image generation network based on the collaborative diffusion model.
[0023] As a further optimization scheme of the face image privacy protection method based on the collaborative diffusion model of the present invention, the detailed steps of step 3.2) are as follows:
[0024] Step 3.2.1), for the identity conditional sub-diffusion model:
[0025] Step 3.2.1.1), construct the identity conditional reconstruction loss L simple_id :
[0026]
[0027] Among them, z0 is the code obtained by inputting each face image in the face image dataset into the encoder network E, and z t is the result of adding noise to z0 in the forward process until t steps, ε is the added noise, c id It is the 512-dimensional face identity tensor obtained by inputting each face image in the face image dataset into the identity information extractor f1. is the noise predicted by the identity conditional sub-diffusion model, θ1 is the parameter of the identity conditional sub-diffusion model; function is at t, z0, ε and c id Under the joint distribution of , the expected square error between the added noise and the predicted noise is calculated;
[0028] Step 3.2.1.2), construct the identity loss L id :
[0029]
[0030] Where D is the decoder network of the variational autoencoder model, is the reconstructed potential representation obtained by the reverse process of the identity conditional diffusion model. The function cos represents the cosine distance between two vectors. To reconstruct the face identity tensor of the image, Simplified notation
[0031] Step 3.2.1.3), construct the overall loss L of the identity conditional diffusion model total :
[0032] L total =L simple_id +λL id ;
[0033] Among them, λ is the preset loss function weight parameter;
[0034] Step 3.2.2), construct the facial structure conditional reconstruction loss L for the facial structure conditional sub-diffusion model simple_mask :
[0035]
[0036] Among them, c mask It is the 19×1024-dimensional facial structure tensor obtained by inputting each face image in the face image dataset into the facial structure information extractor f2. is the noise predicted by the facial structure conditional sub-diffusion model, θ2 is the parameter of the facial structure conditional sub-diffusion model; function is at t, z0, ε and c mask Under the joint distribution of , the expected square error between the added noise and the predicted noise is calculated;
[0037] Step 3.2.3), construct the text conditional reconstruction loss L for the text conditional sub-diffusion model simple_text :
[0038]
[0039] Among them, c text It is the 77×640-dimensional image content tensor obtained by inputting each face image in the face image dataset into the image content information extractor f3. is the noise predicted by the text conditional sub-diffusion model, θ3 is the parameter of the text conditional sub-diffusion model; function is at t, z0, ε and c text Under the joint distribution of , the expected square error between the added noise and the predicted noise is calculated;
[0040] Step 3.2.4), for the cooperative diffusion model, the loss is as follows:
[0041]
[0042] Among them, ε pred,t is the noise predicted by the cooperative diffusion model; the function is at t, z0, ε and [c id ,c mask ,c text ] under the joint distribution of .
[0043] As a further optimization scheme of the face image privacy protection method based on the collaborative diffusion model of the present invention, the detailed steps of step 3.3) are as follows:
[0044] Step 3.3.1), according to the loss function L total The identity conditional sub-diffusion model is trained to obtain a trained identity conditional sub-diffusion model;
[0045] Step 3.3.1.1), input each image in the preprocessed face image dataset into the encoder network E and the identity information extractor f1 to obtain the corresponding z0, c id ;
[0046] Step 3.3.1.2), according to the preset variance scheduling parameter list [β1,β2,…,β t ,…,β T ] Perform a forward process on z0 to obtain the potential representation z with noise T and noise ε, let T be the maximum step set, and the potential representation at each time t is z t :
[0047]
[0048] Among them, α t =1-β t and
[0049] Step 3.3.1.3), transform time t and potential representation z t , noise ε and face identity tensor c id The input identity conditional diffusion model is used to perform the reverse process to obtain the predicted noise and reconstruct the latent representation
[0050]
[0051] Step 3.3.4), combine z0, ε, c id and According to the loss function Ltotal The identity conditional sub-diffusion model is trained to obtain a trained identity conditional sub-diffusion model;
[0052] Step 3.3.2), according to the loss function L simple_mask Training the facial structure conditional sub-diffusion model to obtain a trained facial structure conditional sub-diffusion model;
[0053] Step 3.3.2.1), input each image in the preprocessed face image dataset into the encoder network E and the facial structure information extractor f2 to obtain the corresponding z0, c mask ;
[0054] Step 3.3.2.2), according to the preset variance scheduling parameter list [β1,β2,…,β t ,…,β T ] Perform a forward process on z0 to obtain the potential representation z with noise T and noise ε, T is the maximum step set, and the potential representation of y at each moment is z t :
[0055]
[0056] Among them, α t =1-β t and
[0057] Step 3.3.2.3), transform time t and potential representation z t , noise ε and facial structure tensor c mask The predicted noise is obtained by inverse process of the facial structure conditional diffusion model.
[0058] Step 3.3.2.4), combine z0, ε, and c mask , according to the loss function L simple_mask Training the facial structure conditional sub-diffusion model to obtain a trained facial structure conditional sub-diffusion model;
[0059] Step 3.3.3), according to the loss function L simple_text Training the text conditional sub-diffusion model to obtain a trained text conditional sub-diffusion model;
[0060] Step 3.3.3.1), input each image in the preprocessed face image dataset into the encoder network E and the image content information extractor f3 to obtain the corresponding z0, c text ;
[0061] Step 3.3.3.2), according to the set variance scheduling parameter list [β1,β2,…,β t ,…,β T ] Perform a forward process on z0 to obtain the noisy potential representation z T and noise ε, T is the maximum step set, and the potential representation at each time t is z t :
[0062]
[0063] Among them, α t =1-β t and
[0064] Step 3.3.3.3), transform time t and potential representation z t , noise ε and image content tensor c text The text conditional diffusion model is passed in and the reverse process is performed to obtain the predicted noise
[0065] Step 3.3.3.4), combine z0, ε, and c text , according to the loss function L simple_ttxt Training the text conditional sub-diffusion model to obtain a trained text conditional sub-diffusion model;
[0066] Step 3.3.4), using the trained identity conditional sub-diffusion model, facial structure conditional sub-diffusion model, and text conditional sub-diffusion model, train the collaborative diffusion model according to the loss function L to obtain a trained collaborative diffusion model;
[0067] Step 3.3.4.1), input each image in the preprocessed face image dataset into the encoder network E, identity information extractor f1, facial structure information extractor f2, and image content information extractor f3 to obtain the corresponding z0, c id 、c mask 、c text ;
[0068] Step 3.3.4.2), according to the set variance scheduling parameter list [β1,β2,…,β t ,…,β T ] Perform a forward process on z0 to obtain the potential representation z with noise T and noise ε, T is the maximum step set, and the potential representation at each time t is z t :
[0069]
[0070] Among them, α t=1-β t and
[0071] Step 3.3.4.3), construct three network structures as the dynamic scheduler of U-Net Distribute the noise weights I of the trained identity conditional sub-diffusion model, facial structure conditional sub-diffusion model, and text conditional sub-diffusion model 1,t ,I 2,t ,I 3,t , the formula is as follows:
[0072]
[0073] Step 3.3.4.4), weight normalization, the formula is as follows:
[0074]
[0075] Among them, p represents each pixel position;
[0076] Step 3.3.4.5), transform time t and potential representation z t , noise ε, c id 、c mask 、c text The trained identity conditional sub-diffusion model, facial structure conditional sub-diffusion model, and text conditional sub-diffusion model are respectively passed in to obtain the predicted noise
[0077] Step 3.3.4.6), perform weighted averaging and calculate the prediction result ε at time t pred,t ,as follows:
[0078]
[0079] Step 3.3.4.7), combine z0, ε, ε pred,t 、c id 、c mask 、c text , according to the loss function L, the face image generation network based on the collaborative diffusion model is trained to obtain a trained face image generation network.
[0080] As a further optimization scheme of the face image privacy protection method based on the collaborative diffusion model of the present invention, the specific steps of step 4) are as follows:
[0081] Step 4.1), the size of the image I to be protected for face image privacy is processed into 512×512, and the pixels are normalized; the background mask matrix B of the image is segmented using the BiSeNet network; if the value in B is 1, it means that the pixel here is the background;
[0082] Step 4.2), if identity protection is required, the image I and the preset identity tensor c i ' d Input the trained face image generation network to obtain the protected image I′;
[0083] Step 4.3), if attribute protection is required, encode the preset attribute text into a text tensor c t ' ext , then the image I and the new text tensor c t ' ext Input the trained face image generation network to obtain the protected image I′;
[0084] Step 4.4), finally, background fusion is used to replace the background of the image I to be protected with the protected image. The formula is as follows:
[0085] I′=(1-B)·I′+B·I.
[0086] The present invention adopts the above technical solution and has the following technical effects:
[0087] Based on the existing anti-screen capture watermark, the watermark embedding operation is migrated from the management center to the screen end, realizing the screen-end embedding of the anti-screen capture watermark, reducing the computing and communication burden of the management center, and significantly improving the scalability of the system. BRIEF DESCRIPTION OF THE DRAWINGS
[0088] Figure 1 It is a schematic diagram of the solution flow of the present invention;
[0089] Figure 2 is a schematic diagram of the identity sub-diffusion model of the present invention;
[0090] Figure 3 It is a schematic diagram of the cooperative diffusion model of the present invention. DETAILED DESCRIPTION
[0091] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. To make the above-mentioned objects, features and advantages of the present invention more clearly understandable, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0092] The present invention discloses a face image privacy protection method based on a collaborative diffusion model. Figure 1 As shown, the following steps are included:
[0093] Step 1), select a face image dataset, process the face images therein into the same size of 512×512, and perform pixel normalization on the face images;
[0094] Step 2), constructing a face identity information extractor, a face structure information extractor, and an image content information extractor according to the face image identity, face structure, and image content;
[0095] Step 3), construct and train a face image generation network;
[0096] Step 3.1), construct a face image generation network based on the collaborative diffusion model;
[0097] Step 3.2), construct the loss function for face image generation;
[0098] Step 3.3), input each image in the preprocessed face image data set into the face identity information extractor, the facial structure information extractor, and the image content information extractor, and complete the training in combination with the loss function generated by the face image to obtain a trained face image generation network;
[0099] Step 4), input the image that needs to be protected for face image privacy into the trained face image generation network to obtain a protected image.
[0100] The detailed steps of step 1) are as follows:
[0101] First, select a suitable dataset, such as the CelebA-HQ dataset; then, process the face images in it into the same size of 512×512; finally, perform pixel normalization on the face images.
[0102] The detailed steps of step 2) are as follows:
[0103] First, construct the face identity information extractor f1 of the Inception network;
[0104] Then, the constructed network structure is the facial structure information extractor f2 of the BiSeNet network;
[0105] Finally, if the face image dataset is an image dataset, the network structure constructed is the image content information extractor of the Vision Transformer network for encoding the image data; if the face image dataset is a picture-text pair dataset, the network structure constructed is the image content information extractor of the Transformer network for encoding the text data; let the constructed image content information extractor be f3.
[0106] The detailed steps of step 3.1) are as follows:
[0107] First, a variational autoencoder model is constructed. The variational autoencoder model consists of an encoder network E and a decoder network D. The encoder network E compresses each face image in the face image dataset into a latent space representation z, and then the decoder network D reconstructs the latent space representation z into a reconstructed image. The network uses preset network parameters and does not require training.
[0108] Then, an identity conditional sub-diffusion model is constructed. The identity conditional sub-diffusion model guides the generation of faces by introducing the face identity tensor in the generation process, and its network structure is conditional U-Net; the initial learning rate of the network is 0.000005, the number of iterations is 150, and the batch size is 8; then, a facial structure conditional sub-diffusion model is constructed. The facial structure conditional diffusion model guides the generation of faces by introducing the facial structure tensor in the generation process, and its network structure is conditional U-Net; the initial learning rate of the network is 0.000005, the number of iterations is 200, and the batch size is 8; secondly, a text conditional sub-diffusion model is constructed. The text conditional diffusion model guides the generation of faces by introducing the image content tensor in the generation process, and its network structure is conditional U-Net; the initial learning rate of the network is 0.000005, the number of iterations is 250, and the batch size is 8; finally, a collaborative diffusion model is constructed. The collaborative diffusion model generates faces by weighted averaging the identity conditional sub-diffusion model, the facial structure conditional sub-diffusion model, and the text conditional sub-diffusion model through a dynamic scheduler. The initial learning rate of the network is 0.000005, the number of iterations is 300, and the batch size is 8;
[0109] The detailed steps of step 3.2) are as follows:
[0110] For the identity conditional sub-diffusion model, we first construct the identity conditional reconstruction loss L simple_id :
[0111]
[0112] Among them, z0 is the code obtained by inputting each face image in the face image dataset into the encoder network E, and z t is the result of adding noise to z0 in the forward process until t steps, ε is the added noise, c id It is the 512-dimensional face identity tensor obtained by inputting each face image in the face image dataset into the identity information extractor f1. is the noise predicted by the identity conditional sub-diffusion model, θ1 is the parameter of the identity conditional sub-diffusion model; function is at t, z0, ε and c id Under the joint distribution of , the expected square error between the added noise and the predicted noise is calculated;
[0113] Second, construct the identity loss L id :
[0114]
[0115] Where D is the decoder network of the variational autoencoder model, is the reconstructed potential representation obtained by the reverse process of the identity conditional diffusion model. The function cos represents the cosine distance between two vectors. To reconstruct the face identity tensor of the image, Simplified notation
[0116] Finally, the overall loss L of the identity conditional diffusion model is constructed total :
[0117] L total =L simple_id +λL id ;
[0118] Among them, λ is the preset loss function weight parameter;
[0119] For the facial structure conditional sub-diffusion model, construct the facial structure conditional reconstruction loss L simple_mask :
[0120]
[0121] Among them, c mask It is the 19×1024-dimensional facial structure tensor obtained by inputting each face image in the face image dataset into the facial structure information extractor f2. is the noise predicted by the facial structure conditional sub-diffusion model, θ2 is the parameter of the facial structure conditional sub-diffusion model; function is at t, z0, ε and c mask Under the joint distribution of , the expected square error between the added noise and the predicted noise is calculated;
[0122] For the text conditional sub-diffusion model, construct the text conditional reconstruction loss L simple_text :
[0123]
[0124] Among them, c text It is the 77×640-dimensional image content tensor obtained by inputting each face image in the face image dataset into the image content information extractor f3. is the noise predicted by the text conditional sub-diffusion model, θ3 is the parameter of the text conditional sub-diffusion model; function is at t, z0, ε and ctext Under the joint distribution of , the expected square error between the added noise and the predicted noise is calculated;
[0125] For the cooperative diffusion model, the loss is as follows:
[0126]
[0127] Among them, ε pred,t is the noise predicted by the cooperative diffusion model; the function is at t, z0, ε and [c id ,c mask ,c text ] under the joint distribution of .
[0128] The detailed steps of step 3.3) are as follows:
[0129] like Figure 2 As shown, according to the loss function L total The identity conditional sub-diffusion model is trained to obtain a trained identity conditional sub-diffusion model;
[0130] First, each image in the preprocessed face image dataset is input into the encoder network E and the identity information extractor f1 to obtain the corresponding z0, c id ;
[0131] Secondly, according to the preset variance scheduling parameter list [β1,β2,…,β t ,…,β T ] Perform a forward process on z0 to obtain the potential representation z with noise T and noise ε, let T be the maximum step set, T is set to 1000, β1 is set to 0.0015, β T Set to 0.0195, the variance scheduling parameter list grows linearly, and the potential representation at each time t is z t :
[0132]
[0133] Among them, α t =1-β t and
[0134] Then, the time t, the potential representation z t , noise ε and face identity tensor c id The input identity conditional diffusion model is used to perform the reverse process to obtain the predicted noise and reconstruct the latent representation
[0135]
[0136] Finally, combining z0, ε, c id and According to the loss function L total The identity conditional sub-diffusion model is trained to obtain a trained identity conditional sub-diffusion model;
[0137] According to the loss function L simple_mask Training the facial structure conditional sub-diffusion model to obtain a trained facial structure conditional sub-diffusion model;
[0138] First, each image in the preprocessed face image dataset is input into the encoder network E and the facial structure information extractor f2 to obtain the corresponding z0, c mask ;
[0139] Secondly, according to the preset variance scheduling parameter list [β1,β2,…,β t ,…,β T ] Perform a forward process on z0 to obtain the potential representation z with noise T and noise ε, T is the maximum step set, T is set to 1000, β1 is set to 0.0015, β T Set to 0.0195, the variance scheduling parameter list grows linearly, and the potential representation at each time t is z t :
[0140]
[0141] Among them, α t =1-β t and
[0142] Then, the time t, the potential representation z t , noise ε and facial structure tensor c mask The predicted noise is obtained by inverse process of the facial structure conditional diffusion model.
[0143] Finally, combining z0, ε, and c mask , according to the loss function L simple_mask Training the facial structure conditional sub-diffusion model to obtain a trained facial structure conditional sub-diffusion model;
[0144] According to the loss function L simple_text Training the text conditional sub-diffusion model to obtain a trained text conditional sub-diffusion model;
[0145] First, each image in the preprocessed face image dataset is input into the encoder network E and the image content information extractor f3 to obtain the corresponding z0, c text ;
[0146] Secondly, according to the set variance scheduling parameter list [β1,β2,…,β t ,…,β T ] Perform a forward process on z0 to obtain the noisy potential representation z T and noise ε, T is the maximum step set, T is set to 1000, β1 is set to 0.0015, β T Set to 0.0195, the variance scheduling parameter list grows linearly, and the potential representation at each time t is z t :
[0147]
[0148] Among them, α t =1-β t and
[0149] Then, the time t, the potential representation z t , noise ε and image content tensor c text The text conditional diffusion model is passed in and the reverse process is performed to obtain the predicted noise
[0150] Finally, combining z0, ε, and c text , according to the loss function L simple_text Training the text conditional sub-diffusion model to obtain a trained text conditional sub-diffusion model;
[0151] like Figure 3 As shown, the trained identity conditional sub-diffusion model, the facial structure conditional sub-diffusion model, and the text conditional sub-diffusion model are used to train the collaborative diffusion model according to the loss function L to obtain a trained collaborative diffusion model;
[0152] First, each image in the preprocessed face image dataset is input into the encoder network E, the identity information extractor f1, the facial structure information extractor f2, and the image content information extractor f3 to obtain the corresponding z0, c id 、c mask 、c text ;
[0153] Secondly, according to the set variance scheduling parameter list [β1,β2,…,β t ,…,β T ] Perform a forward process on z0 to obtain the potential representation z with noise Tand noise ε, T is the maximum step set, T is set to 1000, β1 is set to 0.0015, β t Set to 0.0195, the variance scheduling parameter list grows linearly, and the potential representation at each time t is z t :
[0154]
[0155] Among them, α t =1-β t and
[0156] Then, construct three network structures as the dynamic scheduler of U-Net Distribute the noise weights I of the trained identity conditional sub-diffusion model, facial structure conditional sub-diffusion model, and text conditional sub-diffusion model 1,t ,I 2,t ,I 2,t , the formula is as follows:
[0157]
[0158] The weight normalization formula is as follows:
[0159]
[0160] Among them, p represents each pixel position;
[0161] Next, we transform the time t and the potential representation z t , noise ε, c id 、c mask 、c text The trained identity conditional sub-diffusion model, facial structure conditional sub-diffusion model, and text conditional sub-diffusion model are respectively passed in to obtain the predicted noise
[0162] Perform weighted averaging and calculate the prediction result ε at time t pred,t ,as follows:
[0163]
[0164] Finally, combining z0, ε, ε pred,t 、c id 、c mask and c text , according to the loss function L, the face image generation network based on the collaborative diffusion model is trained to obtain a trained face image generation network.
[0165] The specific steps of step 4) are as follows:
[0166] The image I that needs to be protected for face privacy is resized to 512×512 and its pixels are normalized. The background mask matrix B of the image is segmented using the BiSeNet network. If the value in B is 1, it means that the pixel here is the background. If identity protection is required, a new identity tensor c needs to be prepared first. i ' d , and then the image I and the new identity tensor c i ' d Input the trained cooperative diffusion model to obtain the protected image I′; if attribute protection is required, encode the preset attribute text into a text tensor c t ' ext , then the image I and the new text tensor c t ' ext Input the trained cooperative diffusion model to obtain the protection image I′; finally, use background fusion to replace the background of the image I to be protected with the protection image. The formula is as follows:
[0167] I′=(1-B)·I′+B·I.
[0168] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.
[0169] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A face image privacy protection method based on a collaborative diffusion model, characterized in that: The following steps are involved: Step 1), select a face image dataset, process the face images therein into the same size of 512×512, and perform pixel normalization on the face images; Step 2), constructing a face identity information extractor, a face structure information extractor, and an image content information extractor according to the face image identity, face structure, and image content; Step 3), construct and train a face image generation network; Step 3.1), construct a face image generation network based on the collaborative diffusion model; Step 3.2), construct the loss function for face image generation; Step 3.3), input each image in the preprocessed face image data set into the face identity information extractor, the facial structure information extractor, and the image content information extractor, and complete the training in combination with the loss function generated by the face image to obtain a trained face image generation network; Step 4), input the image that needs to be protected for face image privacy into the trained face image generation network to obtain a protected image.
2. The face image privacy protection method based on the collaborative diffusion model as claimed in claim 1, characterized in that: The specific steps of step 2) are as follows: Step 2.1), construct the face identity information extractor f1 of the Inception network; Step 2.2), construct the facial structure information extractor f2 of the BiSeNet network; Step 2.3), if the face image dataset is an image dataset, the network structure constructed is the image content information extractor of the Vision Transformer network for encoding the image data; if the face image dataset is a picture-text pair dataset, the network structure constructed is the image content information extractor of the Transformer network for encoding the text data; let the constructed image content information extractor be f3.
3. The face image privacy protection method based on the collaborative diffusion model as claimed in claim 2, characterized in that: The detailed steps of step 3.1) are as follows: Step 3.1.1), construct a variational autoencoder model, which consists of an encoder network E and a decoder network D. The encoder network E compresses each face image in the face image dataset into a latent space representation z, and then the decoder network D reconstructs the latent space representation z into a reconstructed image; Step 3.1.2), construct an identity conditional sub-diffusion model. The identity conditional sub-diffusion model guides the generation of faces by introducing face identity tensors in the generation process. Its network structure is conditional U-Net. Step 3.1.3), construct a facial structure conditional diffusion model. The facial structure conditional diffusion model guides the generation of faces by introducing facial structure tensors in the generation process. Its network structure is a conditional U-Net. Step 3.1.4), construct a text conditional sub-diffusion model. The text conditional diffusion model guides the generation of faces by introducing image content tensors in the generation process. Its network structure is conditional U-Net. Step 3.1.5), construct a collaborative diffusion model. The collaborative diffusion model uses a dynamic scheduler to perform weighted averaging on the identity conditional sub-diffusion model, the facial structure conditional sub-diffusion model, and the text conditional sub-diffusion model to form a face image generation network based on the collaborative diffusion model.
4. The face image privacy protection method based on the collaborative diffusion model as claimed in claim 3, characterized in that: The detailed steps of step 3.2) are as follows: Step 3.2.1), for the identity conditional sub-diffusion model: Step 3.2.1.1), construct the identity conditional reconstruction loss L simple_id : Among them, z0 is the code obtained by inputting each face image in the face image dataset into the encoder network E, and z t is the result of adding noise to z0 in the forward process until t steps, ε is the added noise, c id It is the 512-dimensional face identity tensor obtained by inputting each face image in the face image dataset into the identity information extractor f1. is the noise predicted by the identity conditional sub-diffusion model, θ1 is the parameter of the identity conditional sub-diffusion model; function is at t, z0, ε and c id Under the joint distribution of , the expected square error between the added noise and the predicted noise is calculated; Step 3.2.1.2), construct the identity loss L id : Where D is the decoder network of the variational autoencoder model, is the reconstructed potential representation obtained by the reverse process of the identity conditional diffusion model. The function cos represents the cosine distance between two vectors. To reconstruct the face identity tensor of the image, Simplified notation Step 3.2.1.3), construct the overall loss L of the identity conditional diffusion model total : L total =L simple_id +λL id ; Among them, λ is the preset loss function weight parameter; Step 3.2.2), construct the facial structure conditional reconstruction loss L for the facial structure conditional sub-diffusion model simple_mask : Among them, c mask It is the 19×1024-dimensional facial structure tensor obtained by inputting each face image in the face image dataset into the facial structure information extractor f2. is the noise predicted by the facial structure conditional sub-diffusion model, θ2 is the parameter of the facial structure conditional sub-diffusion model; function is at t, z0, ε and c mask Under the joint distribution of , the expected square error between the added noise and the predicted noise is calculated; Step 3.2.3), construct the text conditional reconstruction loss L for the text conditional sub-diffusion model simple_text : Among them, c text It is the 77×640-dimensional image content tensor obtained by inputting each face image in the face image dataset into the image content information extractor f3. is the noise predicted by the text conditional sub-diffusion model, θ3 is the parameter of the text conditional sub-diffusion model; function is at t, z0, ε and c text Under the joint distribution of , the expected square error between the added noise and the predicted noise is calculated; Step 3.2.4), for the cooperative diffusion model, the loss is as follows: Among them, ε pred,t is the noise predicted by the cooperative diffusion model; the function is at t, z0, ε and [c id ,c mask ,c text ] under the joint distribution of .
5. The face image privacy protection method based on the collaborative diffusion model as claimed in claim 4, characterized in that: The detailed steps of step 3.3) are as follows: Step 3.3.1), according to the loss function L total The identity conditional sub-diffusion model is trained to obtain a trained identity conditional sub-diffusion model; Step 3.3.1.1), input each image in the preprocessed face image dataset into the encoder network E and the identity information extractor f1 to obtain the corresponding z0, c id ; Step 3.3.1.2), according to the preset variance scheduling parameter list [β1,β2,…,β t ,…,β T ] Perform a forward process on z0 to obtain the potential representation z with noise T and noise ε, let T be the maximum step set, and the potential representation at each time t is z t : Among them, α t =1-β t and Step 3.3.1.3), transform time t and potential representation z t , noise ε and face identity tensor c id The input identity conditional diffusion model is used to perform the reverse process to obtain the predicted noise and reconstruct the latent representation Step 3.3.4), combine z0, ε, c id and According to the loss function L total The identity conditional sub-diffusion model is trained to obtain a trained identity conditional sub-diffusion model; Step 3.3.2), according to the loss function L simple_mask Training the facial structure conditional sub-diffusion model to obtain a trained facial structure conditional sub-diffusion model; Step 3.3.2.1), input each image in the preprocessed face image dataset into the encoder network E and the facial structure information extractor f2 to obtain the corresponding z0, c mask ; Step 3.3.2.2), according to the preset variance scheduling parameter list [β1,β2,…,β t ,…,β T ] Perform a forward process on z0 to obtain the potential representation z with noise T and noise ε, T is the maximum step set, and the potential representation at each time t is z t : Among them, α t =1-β t And Step 3.3.2.3), transform time t and potential representation z t , noise ε and facial structure tensor c mask The predicted noise is obtained by inverse process of the facial structure conditional diffusion model. Step 3.3.2.4), combine z0, ε, and c mask , according to the loss function L simple_mask Training the facial structure conditional sub-diffusion model to obtain a trained facial structure conditional sub-diffusion model; Step 3.3.3), according to the loss function L simple_text Training the text conditional sub-diffusion model to obtain a trained text conditional sub-diffusion model; Step 3.3.3.1), input each image in the preprocessed face image dataset into the encoder network E and the image content information extractor f3 to obtain the corresponding z0, c text ; Step 3.3.3.2), according to the set variance scheduling parameter list [β1,β2,…,β t ,…,β T ] Perform a forward process on z0 to obtain the noisy potential representation z T and noise ε, T is the maximum step set, and the potential representation at each time t is z t : Among them, α t =1-β t And Step 3.3.3.3), transform time t and potential representation z t , noise ε and image content tensor c text The text conditional diffusion model is passed in for the reverse process to obtain the predicted noise Step 3.3.3.4), combine z0, ε, and c text , according to the loss function L simple_text Training the text conditional sub-diffusion model to obtain a trained text conditional sub-diffusion model; Step 3.3.4), using the trained identity conditional sub-diffusion model, facial structure conditional sub-diffusion model, and text conditional sub-diffusion model, the collaborative diffusion model is trained according to the loss function L to obtain a trained collaborative diffusion model; Step 3.3.4.1), input each image in the preprocessed face image dataset into the encoder network E, identity information extractor f1, facial structure information extractor f2, and image content information extractor f3 to obtain the corresponding z0, c id 、c mask 、c text ; Step 3.3.4.2), according to the set variance scheduling parameter list [β1,β2,…,β t ,…,β T ] Perform a forward process on z0 to obtain the potential representation z with noise T and noise ε, T is the maximum step set, and the potential representation at each time t is z t : Among them, α t =1-β t and Step 3.3.4.3), construct three network structures as the dynamic scheduler of U-Net Distribute the noise weights I of the trained identity conditional sub-diffusion model, facial structure conditional sub-diffusion model, and text conditional sub-diffusion model 1,t ,I 2,t ,I 3,t , the formula is as follows: Step 3.3.4.4), weight normalization, the formula is as follows: Among them, p represents each pixel position; Step 3.3.4.5), transform time t and potential representation z t , noise ε, c id 、c mask 、c text The trained identity conditional sub-diffusion model, facial structure conditional sub-diffusion model, and text conditional sub-diffusion model are respectively passed in to obtain the predicted noise Step 3.3.4.6), perform weighted averaging and calculate the prediction result ε at time t pred,t ,as follows: Step 3.3.4.7), combine z0, ε, ε pred,t 、c id 、c mask 、c text , according to the loss function L, the face image generation network based on the collaborative diffusion model is trained to obtain a trained face image generation network.
6. The face image privacy protection method based on the collaborative diffusion model as claimed in claim 5, characterized in that: The specific steps of step 4) are as follows: Step 4.1), the size of the image I to be protected for face image privacy is processed into 512×512, and the pixels are normalized; Use the BiSeNet network to segment the background mask matrix B of the image; if the value in B is 1, it means that the pixel here is the background; Step 4.2), if identity protection is required, the image I and the preset identity tensor c i ′ d Input the trained face image generation network to obtain the protected image I′; Step 4.3), if attribute protection is required, encode the preset attribute text into a text tensor c t ′ ext , then the image I and the new text tensor c t ′ ext Input the trained face image generation network to obtain the protection image I ′ ; Step 4.4), finally, background fusion is used to replace the background of the image I to be protected with the protected image. The formula is as follows: I′=(1-B)·I′+B·I.
Citation Information
Patent Citations
Face identity confrontation recognition privacy protection method and system based on diffusion model
CN117894085A
Method for modifying facial image to be temporally consistent through separated image encoding
WO2024128390A1
