Prompt-independent image protection method for implicit diffusion model

Abstract

The invention discloses a cue word-independent image protection method for an implicit diffusion model, which comprises the following steps of: designing and optimizing a protection loss function of an image in the implicit diffusion model, namely, simultaneously disturbing a mean value and a variance of Gaussian distribution defined by a visual encoder, so that the implicit diffusion model cannot learn semantic information of the image; therefore, the purpose of protecting the image is achieved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of image processing technology in artificial intelligence, and relates to image privacy and copyright protection technology, in particular to a prompt word independent image protection method for implicit diffusion model, which is an image protection method for large-scale implicit diffusion model based on adding micro-noise to disturb the image implicit representation to achieve that the image content cannot be learned by deep learning algorithm. BACKGROUND

[0002] In recent years, deep learning technology, especially image generation model, has developed rapidly. From the earliest generative adversarial network (GAN) to the denoising diffusion probabilistic model (DDPM) proposed by Jonathan Ho et al. in 2020, and a series of implicit diffusion models proposed in 2021 and 2022, such as DALL-E and Stable Diffusion, the images generated by artificial intelligence show high quality and authenticity. In 2017, DeepFake technology was proposed, which is based on GAN to realize the learning and re-creation of human face photos. Since the computing overhead required by DeepFake is large, and it is difficult to achieve high-quality imitation and editing of human faces through a small amount of pictures. However, with the proposal and publicization of large-scale implicit diffusion models, for example, anyone can obtain the latest weights of the high-performance implicit diffusion model Stable Diffusion for free on HuggingFace.con, and the proposal of small sample-oriented implicit diffusion model fine-tuning technology (Textual Inversion, DreamBooth), malicious destroyers can complete the fine learning of the human face / painting style contained in the image under the restriction of a small amount of data, such as 4-5 pictures containing the same human face / painting style, and limited computing resources (single graphics card). After learning, based on the existing ability of large-scale implicit diffusion model, a large number of infringing pictures can be generated to imitate and cause great harm to personal privacy or copyright.

[0003] To protect image data from being misused by implicit diffusion model abusers, existing works propose two technical routes for data protection by adding micro-noise. Technical route one: the core idea of this technical route is to use the concept of adversarial examples in the field of machine learning. The representative method is the AdvDM algorithm proposed by Chumeng Liang et al. in 2023. Adversarial examples deviate from the normal data distribution and represent the worst-case performance of the model. By implanting adversarial noise that can greatly change the behavior of the neural network into normal data, it becomes an adversarial example, so that the model cannot converge on these data during training, and thus cannot learn the natural features in the data, achieving the purpose of protecting image privacy and copyright. Technical route two: the core idea of this technical route is to use the concept of unlearnable examples in the field of machine learning. The representative method is the ASPL (Alternating Surrogate and Perturbations Learning) algorithm proposed by Thanh Van Le et al. in 2023. Unlearnable examples are generated for the training process of the model, which induces the model to learn false correlation features in the image, forcing the model to converge to a globally poor solution. This technical route combines the concept of unlearnable examples with implicit diffusion model fine-tuning algorithms such as DreamBooth, making it effective for this type of image generation model. Recently, Boyang Zheng et al. proposed that the two seemingly conflicting technical routes can be integrated to improve performance in some scenarios.

[0004] Both of the above technical routes have achieved success in designing image protection micro-noise for implicit diffusion models, but the above work is based on the unrealistic assumption that the image protector is aware of the text prompt used by the image abuser when fine-tuning the model, and protects against this prompt. When the image protector does not know the text prompt used by the image abuser, the performance of the above techniques for image protection will be greatly reduced. At the same time, the above scheme does not fully consider the impact of the visual encoder, which is unrelated to the prompt, in the implicit diffusion model. Existing technology has not been able to generate micro-noise based on the visual encoder, and has not been able to take advantage of the natural independence of the visual encoder and the text encoder, making it difficult to enhance the robustness of image protection micro-noise for implicit diffusion models when facing different prompts, as well as the practicality and reliability of image protection technology. SUMMARY

[0005] To overcome the shortcomings of the prior art, the present invention provides a cue-word-independent image protection method for implicit diffusion models. Based on the design of subtle noise in the visual encoder of the implicit diffusion model, the method aims to perturb the output of the visual encoder of the implicit diffusion model by designing image subtle noise, thereby achieving robust image protection that is immune to interference from fine-tuning cue words used by data abusers. While ensuring that the image does not change significantly to the naked eye, malicious users cannot learn sensitive information from the image by fine-tuning the implicit diffusion model, thereby protecting personal privacy and artist copyright.

[0006] In this invention, because both training and inference of the implicit diffusion model occur in the latent space, when the image representation in the latent space (i.e., the input actually seen by the internal diffusion model) is severely corrupted, the model cannot learn the true characteristics of the image. Furthermore, due to the independence of the visual encoder and the text encoder, the method of this invention is largely unaffected by changes in the prompt words used by malicious abusers.

[0007] The visual encoder of the implicit diffusion model models each input image as a Gaussian distribution in the latent space, and the mean and variance of the Gaussian distribution are given by the output of the visual encoder. We found that the mean of the Gaussian distribution mainly contains the texture information of the image (Texture), while the variance change of the Gaussian distribution has a great damage to the semantic information of the image (Semantic). At the same time, the mean and variance output by the visual encoder have a large difference in value. Combining the above two points, the present invention creatively proposes to optimize the sum of the square difference of the mean and the square difference of the logarithmic variance of the latent space representation of the noise-perturbed image and the original image as the protection loss, thereby simultaneously perturbing the mean and variance of the Gaussian distribution defined by the visual encoder, making it impossible for the implicit diffusion model to learn the semantics contained in the image, thereby achieving the purpose of protecting the image. That is, the present invention obtains protection noise by designing a protection loss function L and minimizing the loss L. This is explained in detail below.

[0008] For the convenience of discussion, we first give a unified explanation of the notations used in this invention. Suppose the clean image x∈[0,255] we want to protect is H×W×C , is an image with a height of H, a width of W, and a number of channels of C, usually a 512×512×3 color image, with each pixel value in the range [0,255]. Let the protection noise we want to add be δ, and for each pixel in the image, the maximum range that the protection noise can disturb is set to ∈, which serves as the upper and lower bounds of the feasible domain of attack intensity. For the requirement that the disturbance (noise) is not perceptible to the naked eye, ∈ is usually set to From this, we can define that the protection noise needs to satisfy two feasible domains: pixel feasible domain, that is, [-x, 255-x] H×W×C, to prevent color overflow and result in inaccurate results; attack strength feasible region, i.e. [-∈,∈] H×W×C , to ensure that the disturbance (noise) is not visible to the naked eye. The resulting protective noise needs to satisfy two feasible regions at the same time, i.e. we need δ∈[-x,255-x] H×W×C ∩[-∈,∈] H×W×C . We define the projection operation (Project) as Project(x, lb, ub) = min(max(x, lb), ub), where lb and ub are formal parameters of the function, indicating that the parameter x is projected to the upper and lower bounds of the interval, i.e. x is projected to the interval [ub, lb]; the min function returns the smaller of the two parameters, and the max function returns the larger of the two parameters. Pixel feasible region: limits the added disturbance to not cause color overflow in the final image. Attack strength feasible region: limits the added disturbance to not exceed the range we specify, and the upper and lower bounds of the range are given by the hyperparameter ∈. The resulting disturbance is in the intersection of the two feasible regions, i.e. two projection operations are needed. Let the visual encoder in the implicit diffusion model be E, which is part of the implicit diffusion model. For input x, the visual encoder E represents it in the latent space as a Gaussian distribution (the latent distribution of the clean picture x). The mean of the distribution is determined by the output of the visual encoder E, denoted as E μ (x); the variance of the distribution is also determined by the output of E, denoted as E σ (x), and finally we denote the Gaussian distribution as Let the protection loss function we want to solve be L, and the input of the function be the clean picture, the current optimized protection disturbance (protection noise), and the visual encoder. Use the projected gradient q descent method (Projected Gradient Descent) to solve it. When updating the protection, there is a hyperparameter α that specifies the step size when updating, which is usually set to one tenth of ∈.

[0009] The technical solutions provided by the present application are as follows:

[0010] An image protection algorithm for implicit diffusion models independent of prompt words, comprising the following steps:

[0011] A. Define the pixel feasible region and define the projection operation; first initialize the protection noise uniformly in [-∈,∈], denoted as δ (0) , and project it into the pixel feasible region.

[0012] B. For the input picture (clean picture) x, repeat the following operations t = 0, 1, 2, …, T-1 times:

[0013] B1. Input the clean picture x into the visual encoder E to obtain the latent distribution of the clean picture In this step, the computation graph is not needed to be generated.

[0014] B2. Update the protection noise δ (t) to the clean picture x, to get the noisy picture (x+δ (t) ); input the noisy picture (x+δ (t) ) into the visual encoder E to get the latent distribution of the perturbed picture In this step, the computation graph is kept for the derivation.

[0015] B3. Calculate the protection loss L(x, δ (t) ; E) = (E μ (x)-E μ (x+δ (t) )) 2 +(logE σ (x)-logE σ (x+δ (t) )) 2 , where the addition, subtraction, square and logarithm operations are all pixel-wise operations.

[0016] In the present application, the protection loss is specifically designed as the sum of the two-norm of the difference of the mean value and the difference of the logarithmic variance. Optimizing the protection loss means that the mean and variance of the protected picture should be as far as possible from the mean and variance of the clean picture, so as to achieve the protection effect.

[0017] B4. Perform back propagation to calculate the derivative and update the protection noise δ as follows: where the addition and multiplication are pixel-wise operations. And it is projected to the pixel feasible region and the attack strength feasible region.

[0018] C. Add the finally obtained protection noise δ (T-1) to the clean picture pixel by pixel to change the clean picture into the protected picture x' = x+δ (T-1) .

[0019] Through the above steps, the image protection for the implicit diffusion model independent of the prompt word can be realized.

[0020] Compared with the prior art, the present application has the beneficial technical effects as follows:

[0021] The existing image protection algorithm for implicit diffusion model is based on the assumption that the protector knows the prompt words used by the malicious user, and its performance is not satisfactory when the malicious user uses different prompt words for fine-tuning training. By using the technical solution of the application, the robustness and security of the protected image in the above scenario can be improved by implanting micro-noise to disturb the mean and variance of the Gaussian distribution corresponding to the hidden space of the protected image, and it can have potential applications in many fields: from the perspective of privacy and copyright protection, the algorithm proposed in the application can protect personal privacy photos published on the network, such as faces, paintings with special styles and patterns, and protect the rights and interests of citizens from being infringed. In the military field, it can be used as a "post-protection" for military sensitive images. By using the technology provided by the application, even if the images are accidentally leaked or stolen, the other party can also prevent the training of implicit diffusion models on these images, thereby maintaining the security of the images. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a flow chart of the implementation of the method provided by the application. DETAILED DESCRIPTION

[0023] The application will be further described by examples in conjunction with the drawings, but the scope of the application is not limited in any way.

[0024] The technical solution of the application for image protection against implicit diffusion model is independent of the prompt word, which disturbs the mean and variance of the Gaussian distribution corresponding to the hidden space of the protected image by implanting micro-noise, prevents the training of implicit diffusion model on these protected images, and improves the robustness and security of the protected image in the above scenario.

[0025] The implementation process of the method provided by the application is shown in Figure 1 The specific implementation is as follows:

[0026] A. Network preparation and data preprocessing: first, select a visual encoder commonly used in implicit diffusion model, such as VAE, and load it onto the GPU. For the image dataset to be protected, scale and normalize each picture in the dataset so that it meets the input requirements of the above visual encoder, and load it onto the corresponding GPU.

[0027] B. The protection noise δ (0) to be added is initialized to the same size as the input picture x, and the value is randomly distributed in the interval [-∈,∈] tensor, and loaded onto the corresponding GPU. Then δ (0) is projected to the pixel feasible region. Next, the protection noise is optimized and solved by iteration t=0,1,2,…,T-1 steps:

[0028] B1. Input the original image x into the visual encoder E and perform forward propagation to obtain the hidden distribution of the original image The computation graph is not needed in this step.

[0029] B2. Input the noisy image (x + δ (t) ) into the visual encoder E and perform forward propagation to obtain the hidden distribution of the perturbed image N(E μ (x + δ (t) ), The computation graph is preserved in this step for derivation.

[0030] B3. Calculate the protection loss L(x, δ (t) ; E) = (E μ (x) - E μ (x + δ (t) )) 2 + (log E σ (x) - log E σ (x + δ (t) )) 2 , where addition, subtraction, squaring, and logarithm operations are all pixel-wise operations.

[0031] B4. Perform backpropagation to calculate the derivative and update the protection noise δ , where addition and multiplication are pixel-wise operations. Project it to the pixel feasible region and the attack strength feasible region.

[0032] C. Add the final protection noise δ (T-1) to the clean image pixel by pixel to make the clean image into a protected image x' = x + δ (T-1) .

[0033] Implementation Example 1:

[0034] Without loss of generality, we randomly select an individual from the CelebA-HQ face dataset, and randomly select four images from all the images of this individual as the image set to be protected to illustrate the actual implementation process of the algorithm. We set the image size to 512x512x3, the perturbation strength range ∈ = 0.05, the number of iterations T = 1000, the update step size α = 0.005, the visual encoder is the AutoEncoderKL used in Stable Diffusion v1.5, which is a specific implementation of VAE, and the weights are obtained from HuggingFace.com. For software, we mainly implement it through python code, mainly using the deep learning library PyTorch and diffusers. For hardware facilities, we use an RTX3090 graphics card for calculation.

[0035] A. First, we load the image using the Image.open() function in PIL's Image library. Then we convert the image to tensor format using torchvision.transform's ToTensor() function, and finally load it into the GPU.

[0036] B. We initialize δ0 using PyTorch's empty_like(x).uniform_(-8 / 255,8 / 255).cuda(). We can find the definition of the visual encoder E model in the diffusers.AutoEncoderKL library, and load the downloaded visual encoder weights into the GPU using the torch.load() function.

[0037] B1. We input the original image x into the visual encoder E and call the encode() function for forward propagation to get the latent distribution of the original image In this step, we prevent the generation of the computation graph by setting torch.no_grad().

[0038] B2. Set δ (t) .requires_grad() to indicate that the computation graph of δ (t) needs to be preserved for backpropagation. Input the noisy image (x+δ (t) ) into the visual encoder E and call the encode() function for forward propagation to get the latent distribution of the perturbed image N(E μ (x+δ (t) ),

[0039] B3. Calculate the protection loss L(x,δ (t) ;E) = (E μ (x)-E μ (x+δ (t) )) 2 +(logE σ (x)-logE σ (x+δ (t) )) 2 , where addition, subtraction, squaring, and logarithm operations are all pixel-wise operations. Here, the mean and variance are obtained from the latent_dist attribute of AutoEncoderKL, which are latent_dist.mean and latent_dist.logvar, respectively.

[0040] B4. Backpropagation is performed using the loss.backward() function to calculate the derivative and the protection noise δ is updated where the addition and multiplication are pixel-wise operations. And project them into the pixel feasible region and the attack strength feasible region. The torch.clamp() function of the PyTorch library is called to realize the projection.

[0041] C. The protection noise is added to the image pixel by pixel to obtain the protected image x' = x + δ (T-1) x' is transferred from the GPU to the CPU, and the transform.ToPILImage() function is used to convert the tensor into a picture format, and the Image.save() function is called to store the picture as a png picture. At this point, the protection noise solved by the algorithm is contained in the saved png, which will prevent the fine-tuning algorithm of the implicit diffusion model from learning useful information therefrom.

[0042] It should be noted that the purpose of publishing the embodiments is to help further understand the present application, but those skilled in the art can understand that various replacements and modifications are possible without departing from the scope of the present application and the appended claims. Therefore, the present application should not be limited to the disclosed embodiments, and the scope of the present application is defined by the scope of the claims.

Claims

1. A method for image protection against implicit diffusion model, prompting word independent, characterized in that, By designing the protection loss function of the image in the implicit diffusion model and optimizing, the sum of the square difference of the mean and the square difference of the logarithmic variance of the corresponding implicit space representation of the original picture and the picture disturbed by noise is optimized as the protection loss, that is, by simultaneously disturbing the mean and the logarithmic variance of the Gaussian distribution defined by the visual encoder, the implicit diffusion model cannot learn the semantic information of the image, so as to achieve the purpose of protecting the image; including the following steps: A. Preprocess the image; uniformly initialize the protection noise of the image, and project it into the pixel feasible region; B. Design the protection loss function of the image in the implicit diffusion model; through multi-step iteration, the protection noise is optimized and solved, and the protection noise is prevented from exceeding the pixel feasible region and the attack strength feasible region, so as to obtain the optimized protection noise; The protection loss function is specifically designed as the sum of the two differences of the mean difference and the logarithmic variance difference; the optimization of the protection loss is to set the solving goal as making the mean and variance of the distribution of the protected picture in the implicit space be farthest from the clean picture; C. Add the final protection noise obtained and the clean picture pixel by pixel to obtain the protected image; Through the above steps, the image protection for the implicit diffusion model independent of the prompt word can be realized.

2. The hint word independent image protection method for implicit diffusion model as claimed in claim 1, wherein, In step A, the clean image to be protected x ∈ [0, 255] H×W×C ; the added protection noise is δ; x is an image with height H, width W, and channel number C; the feasible region of pixels is defined as [-x, 255-x] H×W×C , to prevent color overflow and inaccurate results; the feasible region of attack strength is defined as [-∈, ∈] H×W×C , so that the perturbation noise is not visible to the naked eye; ∈ is the maximum range that the protection noise can be disturbed; when solving, it is necessary to ensure that the protection noise δ is in the intersection of the two feasible regions, i.e. δ ∈ [-x, 255-x] H×W×C ∩ [-∈, ∈] H×W×C .

3. The hint word independent image protection method for implicit diffusion model as claimed in claim 2, wherein, ∈ is set to 4. The hint word independent image protection method for implicit diffusion model as claimed in claim 2, wherein, The protection noise of the image is initialized uniformly on [-∈,∈], denoted as δ (0) and projected into the pixel feasible region.

5. The hint word independent image protection method for implicit diffusion model as claimed in claim 1, wherein, The input of step B for designing the protection loss function L of the image in the implicit diffusion model is the clean picture, the protection noise of the current optimization iteration step and the visual encoder; it is represented as: L(x, δ (t) ; E) = (E μ (x) - E μ (x + δ (t) )) 2 + (log E σ (x) - log E σ (x + δ (t) )) 2 , where the addition, subtraction, squaring, and taking logarithm operations are all pixel-wise operations; E is a visual encoder in the implicit diffusion model; x is an input clean picture; δ (t) is the protected noise for the t-th step iteration; the visual encoder E represents x in the latent space as a Gaussian distribution, i.e., the latent distribution of x, denoted as E μ (x) is the mean of the distribution; E σ (x) is the variance of the distribution.

6. The hint word independent image protection method for implicit diffusion model as claimed in claim 5, wherein, Specifically, a clean picture x is input into a vision encoder E to obtain an implicit distribution of the clean picture The protection noise δ (t) Is added to the clean picture x to obtain a noisy picture (x+δ (t) ); and the noisy picture (x+δ (t) ) is input into the vision encoder E to obtain an implicit distribution of the perturbed picture And the computational graph is retained for derivation.

7. The hint word independent image protection method for implicit diffusion model as claimed in claim 6, wherein, Backpropagation is performed to calculate the derivatives And the protection noise δ is updated and projected to the pixel feasible region and the attack strength feasible region; the protection noise updating method is represented as: Wherein, the addition and multiplication are pixel by pixel operation, and the hyperparameter alpha is used to specify the step length when updating.

8. The hint-independent image protection method for implicit diffusion model according to claim 7, wherein, Step C gives the protected image x by ′ : x ′ = x + δ (T-1) . ​ 9. The image protection method for the implicit diffusion model that is independent of the prompt word as claimed in claim 1, wherein In the network preparation and data preprocessing process, first, select the visual encoder in the implicit diffusion model to load onto the GPU; scale and normalize each picture in the image data set to be protected so that it meets the input requirements of the visual encoder, and load it onto the corresponding GPU; the protection noise to be added is randomly initialized and loaded onto the corresponding GPU.

10. The hint-independent image protection method against implicit diffusion model according to claim 9, wherein, The visual encoder uses the VAE model in the implicit diffusion model: AutoEncoderKL model. ​

Images