Text-to-image diffusion rapid concept erasing method

Abstract

The invention discloses a text-to-image diffusion rapid concept erasing method. The method comprises the following steps: (1) acquiring misleading or false images by using an open source website and preprocessing the misleading or false images; (2) constructing a text-to-image diffusion model: designing a target concept erasing strategy by using a knowledge distillation framework; introducing a calibration set to design other concept retention strategies; (3) generating an image corresponding to the text description; according to the method, the specific concept can be erased efficiently, and meanwhile, the semantic consistency of other related concepts is kept, so that the safety and the reliability of an image generation process are remarkably improved.

Description

Technical Field

[0001] The invention relates to the technical field of text recognition, and in particular to a text-to-image diffusion fast concept erasing method. Background Art

[0002] Text-to-image technology is a major breakthrough in the field of generative models in recent years. By inputting a text description, the text-to-image model is able to generate an image that is closely consistent with the text semantics. This technological advancement not only allows users to create realistic images, but also allows the model to be fine-tuned to meet specific needs, facilitating personalized customization.

[0003] Currently, the mainstream text-to-image concept erasing methods fall into two categories: (1) fine-tuning the weights of the U-Net module in the stable diffusion model through negative guidance to eliminate specific concepts; (2) refining the cross-attention block by adjusting the projection matrix related to the cue, aiming to optimize the cross-attention block for concept removal in U-Net. Despite these advances, these methods focus more on the U-Net module of the stable diffusion model, which usually requires a lot of computing resources, resulting in high computational cost and unstable performance. Summary of the invention

[0004] Purpose of the invention: The purpose of the present invention is to provide a text-to-image diffusion fast concept erasure method to solve the problems of complex calculation, high time cost and unstable effect of existing models.

[0005] Technical solution: The method for rapid concept erasure from text to image diffusion described in the present invention comprises the following steps:

[0006] (1) Using open source websites to obtain misleading or false images and pre-processing them;

[0007] (2) Constructing a text-to-image diffusion model: including designing a target concept erasure strategy by using a knowledge distillation framework; introducing a calibration set to design other concept retention strategies;

[0008] (3) Generate an image corresponding to the text description.

[0009] Furthermore, in step (2), the target concept erasure strategy is designed by using the knowledge distillation framework as follows:

[0010] First, the multimodal pre-trained model CLIP is loaded as the teacher CLIP model and the student CLIP model respectively. At the same time, the concept text to be erased, the empty concept text and other irrelevant concept text are input into the teacher and student CLIP models respectively to obtain the text embedding matrix. The calculation formula is:

[0011] emb=E clip (y)

[0012] Among them, c is the concept embedding matrix, y is the target concept;

[0013] Then, based on the knowledge distillation framework, the model is fine-tuned to make the target concept and the empty concept have the same characteristics in the text space. The calculation formula is:

[0014]

[0015] Among them, emb θ (c) Represent the text embedding of target concept c using the student CLIP model with parameters θ; and The parameters are θ * The frozen teacher CLIP model represents the text embeddings of the null concept c0 and the target concept c, and the coefficients a and β control the contribution of the embedding from the teacher model in the loss function.

[0016] Furthermore, in step (2), the calibration set is introduced to design other concept retention strategies as follows: a concept retention loss function based on the difference between embeddings is designed, and the calculation formula is:

[0017]

[0018] where δ is the regularization coefficient used to balance the trade-off between retaining other concepts and erasing the target concept, c i denotes other concepts that are randomly selected from the calibration set and are different from the target concept c.

[0019] Furthermore, step (3) is specifically as follows: by inputting a text prompt, the model generates an image corresponding to the text description without including the target concept.

[0020] The present invention provides a text-to-image diffusion fast concept erasure system, comprising:

[0021] Image acquisition module: used to obtain misleading or false images from open source websites and perform preprocessing;

[0022] Text-to-image diffusion module: used to build a text-to-image diffusion model: including designing a target concept erasure strategy by using a knowledge distillation framework; introducing a calibration set to design other concept retention strategies;

[0023] Generation module: used to generate images corresponding to text descriptions.

[0024] Furthermore, in the text-to-image diffusion module, the target concept erasure strategy is designed by using the knowledge distillation framework as follows:

[0025] First, the multimodal pre-trained model CLIP is loaded as the teacher CLIP model and the student CLIP model respectively. At the same time, the concept text to be erased, the empty concept text and other irrelevant concept text are input into the teacher and student CLIP models respectively to obtain the text embedding matrix. The calculation formula is:

[0026] emb=E clip (y)

[0027] Among them, c is the concept embedding matrix, y is the target concept;

[0028] Then, based on the knowledge distillation framework, the model is fine-tuned to make the target concept and the empty concept have the same characteristics in the text space. The calculation formula is:

[0029]

[0030] Among them, emb θ (c) Represent the text embedding of target concept c using the student CLIP model with parameters θ; and The parameters are θ * The frozen teacher CLIP model represents the text embeddings of the null concept c0 and the target concept c, and the coefficients a and β control the contribution of the embedding from the teacher model in the loss function.

[0031] Furthermore, in the text-to-image diffusion module, the calibration set is introduced to design other concept retention strategies as follows: a concept retention loss function based on the difference between embeddings is designed, and the calculation formula is:

[0032]

[0033] where δ is the regularization coefficient used to balance the trade-off between retaining other concepts and erasing the target concept, c i denotes other concepts that are randomly selected from the calibration set and are different from the target concept c.

[0034] Furthermore, in the generation module, the details are as follows: by inputting a text prompt, the model generates an image corresponding to the text description without including the target concept.

[0035] An electronic device described in the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, any one of the methods for rapid concept erasure from text to image diffusion is implemented.

[0036] The storage medium of the present invention stores a computer program, and when the computer program is executed by a processor, any one of the methods for rapid concept erasure from text to image diffusion is implemented.

[0037] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: it can efficiently achieve the erasure of specific concepts while retaining the semantic consistency of other related concepts, thereby significantly improving the security and reliability of the image generation process. By adopting the fine-tuning method of knowledge distillation, the present invention not only greatly reduces the training time required for concept erasure to about 15 seconds, but also avoids the modification of complex model structures and reduces the consumption of computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is the overall flow chart of the present invention;

[0039] Figure 2 A diagram showing the results of constructing a calibration set for the present invention;

[0040] Figure 3 It is a rendering of the target removal concept of the present invention. DETAILED DESCRIPTION

[0041] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0042] like Figure 1 As shown, a text-to-image diffusion fast concept erasing method disclosed in an embodiment of the present invention includes the following steps:

[0043] S0, model training phase, by inputting misleading or false data (a piece of text or vocabulary concept);

[0044] S1. Design a target concept erasure strategy by using the knowledge distillation framework, including:

[0045] The multimodal pre-trained model CLIP is loaded as the teacher CLIP model and the student CLIP model respectively. At the same time, the concept text to be erased, the empty concept text and other irrelevant concept text are input into the teacher and student CLIP models respectively to obtain the text embedding matrix. The calculation formula is:

[0046] emb=E clip (y)

[0047] Where c is the concept embedding matrix and y is the target concept. It should be noted that the input text should be consistent with the pre-trained multimodal model CLIP version, which only supports English text concepts;

[0048] Then, based on the knowledge distillation framework, the model is fine-tuned to make the target concept and the empty concept have the same characteristics in the text space. The calculation formula is:

[0049]

[0050] Among them, emb θ (c) Text embedding of target concept c using the student-CLIP model with parameters θ. and The parameters are θ * The frozen teacher CLIP model represents the text embedding of the empty concept c0 and the target concept c, and the coefficients a and β control the contribution of the embedding from the teacher model in the loss function. It should be noted that the shape of all text embedding matrices obtained by the CLIP model should be consistent, and the knowledge distillation method is a model compression technique that can transfer knowledge from a complex teacher model to a smaller student model, enabling the student model to reduce computational costs while maintaining performance;

[0051] S2. Introduce the calibration set to design other concept retention strategies to ensure that other concepts are retained when erasing the target concept, including: by constructing a calibration set, based on the knowledge distillation framework, while the model erases the text embedding features of the template concept, the model's text embedding features on other concepts remain unchanged. This process designs a concept retention loss function based on the difference between embeddings, and the calculation formula is:

[0052]

[0053] where δ is the regularization coefficient used to balance the trade-off between retaining other concepts and erasing the target concept, c i represents other concepts that are different from the target concept c and are randomly selected from the calibration set;

[0054] It should be noted that this function is intended to force the model to ensure the semantic integrity of other concepts through the use of the calibration set and to avoid concept drift in the process of erasing the target concept;

[0055] S3. Finally, we obtain a text-to-image diffusion fast concept erasure model. By inputting text prompts, the model generates images corresponding to the text description without containing the target concept.

[0056] Specific experiments:

[0057] By using multi-field experimental data sets (artistic style data set, ImageNet data set, I2P data set, IP copyright data set), the method provided by the present invention and the effect of its implementation are compared and verified;

[0058] In the multi-domain datasets, the English prompts are unified according to the CLIP version, which are used to input into the model to generate images matching the text. The length of a single English prompt varies and can contain special emoticons; the art style dataset is unified into a specific artist style and other descriptive concepts; the ImageNet dataset consists of a variety of entity class concepts; the I2P dataset contains text data that can induce the text-to-image diffusion model to generate unsafe images; the IP copyright dataset contains text data that can induce the text-to-image diffusion model to generate IP copyright infringement;

[0059] First, determine the target concept to be erased, and then build a model according to the technical solution provided by the present invention;

[0060] Experiment 1: To verify that the present invention is faster and more effective in training speed than the existing mainstream methods. This paper selects three concept erasure models for comparative experiments. These three models are: ESD, which is a SOTA model published in the ICCV 2023 top computer vision conference and can be applied to a variety of fields; UCE, which is another SOTA model provided by the author of ESD and is applicable to related fields of concept erasure; SPM, which is the latest SOTA model published in the CVPR 2024 top computer vision conference and is applicable to a variety of fields;

[0061] Through experiments on erasing the "Van Gogh style", the final comparison results of training time are shown in Table 1:

[0062] Table 1

[0063]

[0064] From the data in the table, we can see that the method of the present invention can erase the target concept in less than 15 seconds by fine-tuning the parameters of the text encoder, which is 50 to 600 times faster than other SOTA models. The main reason is that the current mainstream method fine-tunes the parameters of the U-Net module, which consumes more computing time and resources than fine-tuning the text encoder parameters in the CLIP model.

[0065] Experiment 2: To verify the effectiveness of the present invention in erasing target concepts and retaining other concepts, the present invention selected the multi-domain datasets mentioned above and compared them with the SOTA model in Experiment 1. The experimental results of artistic style erasure are shown in Figure 2. Figure 3 As shown in Table 2, the experimental results of specific content erasure are shown in Table 3, and the experimental results of unsafe concept erasure are shown in Table 4.

[0066] Table 2

[0067]

[0068] Artistic style erasing experiment: In Table 2, CLIP-S measures the match between the generated image and the text description. The higher the score, the better the alignment. VQA-Score reflects the performance of the image in the visual question answering task and evaluates the retention of semantic information. The higher the score, the better the semantic retention. FID measures the visual quality difference between the generated image and the real image. The lower the score, the higher the generation quality. ACC-style is defined as the ratio of the number of images that do not contain the target artistic style to the total number of images. The higher the ratio, the better the style erasing effect.

[0069] As shown in Table 2, in the experiments of erasing Van Gogh style and Monet style, the results of this invention are comparable to those of other SOTA models in terms of CLIP-S, VQA-S, and FID, but the results are the best in terms of the style erasing effect ACC-style. The corresponding images are generated by these models, and the more specific experimental effect diagrams are shown in the figure below. Figure 3 As shown;

[0070] Specific content erasing experiment: In Table 3, each cell represents the probability that the image generated by the erasing method still contains the category after the category is erased. It can be seen that the erasing performance of the present invention on 10 different specific content categories is significantly better than that of the ESD and SPM methods;

[0071] Table 3

[0072]

[0073]

[0074] Unsafe concept erasing experiment: As shown in Table 4, after erasing the concepts of nudity and violence, different models can be verified by generating pictures based on text on the I2P dataset. It can be verified that the present invention also has the ability to erase these unsafe concepts. In addition, since the difficulty of erasing abstract concepts and concrete concepts varies, it makes it more difficult to obtain an accurate representation of the target concept in the text space.

[0075] Table 4

[0076] method Naked erasure accuracy Brute force erasure accuracy ESD 0.994 0.809 UCE 0.987 0.584 SPM 0.975 0.691 The present invention 0.978 0.648

[0077] An embodiment of the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is loaded into the processor, any one of the methods for rapid concept erasure from text to image diffusion is implemented.

[0078] An embodiment of the present invention further provides a storage medium storing a computer program, wherein the computer program, when executed by a processor, implements any one of the methods for rapid concept erasure from text to image diffusion.

Claims

1. A text-to-image diffusion fast concept erasing method, characterized in that: The following steps are involved: (1) Using open source websites to obtain misleading or false images and pre-processing them; (2) Constructing a text-to-image diffusion model: including designing a target concept erasure strategy by using a knowledge distillation framework; introducing a calibration set to design other concept retention strategies; (3) Generate an image corresponding to the text description.

2. The method for rapid concept erasure from text to image diffusion according to claim 1, characterized in that: In step (2), the target concept erasure strategy is designed by using the knowledge distillation framework as follows: First, the multimodal pre-trained model CLIP is loaded as the teacher CLIP model and the student CLIP model respectively. At the same time, the concept text to be erased, the empty concept text and other irrelevant concept text are input into the teacher and student CLIP models respectively to obtain the text embedding matrix. The calculation formula is: emb=E clip (and) Among them, c is the concept embedding matrix, y is the target concept; Then, based on the knowledge distillation framework, the model is fine-tuned to make the target concept and the empty concept have the same characteristics in the text space. The calculation formula is: Among them, emb θ (c) Represent the text embedding of target concept c using the student CLIP model with parameters θ; and The parameters are θ * The frozen teacher CLIP model represents the text embeddings of the null concept c0 and the target concept c, and the coefficients a and β control the contribution of the embedding from the teacher model in the loss function.

3. The method for rapid concept erasure from text to image diffusion according to claim 2, characterized in that: In step (2), the calibration set is introduced to design other concept retention strategies as follows: a concept retention loss function based on the difference between embeddings is designed, and the calculation formula is: where δ is the regularization coefficient used to balance the trade-off between retaining other concepts and erasing the target concept, c i denotes other concepts that are randomly selected from the calibration set and are different from the target concept c.

4. The method for rapid concept erasure from text to image diffusion according to claim 1, characterized in that: Step (3) is as follows: by inputting a text prompt, the model generates an image corresponding to the text description without containing the target concept.

5. A text-to-image diffusion fast concept erasure system, characterized in that: include: Image acquisition module: used to obtain misleading or false images from open source websites and perform preprocessing; Text-to-image diffusion module: used to build a text-to-image diffusion model: including designing a target concept erasure strategy by using a knowledge distillation framework; introducing a calibration set to design other concept retention strategies; Generation module: used to generate images corresponding to text descriptions.

6. A text-to-image diffusion fast concept erasure system according to claim 5, characterized in that: In the text-to-image diffusion module, the target concept erasure strategy is designed by using the knowledge distillation framework as follows: First, the multimodal pre-trained model CLIP is loaded as the teacher CLIP model and the student CLIP model respectively. At the same time, the concept text to be erased, the empty concept text and other irrelevant concept text are input into the teacher and student CLIP models respectively to obtain the text embedding matrix. The calculation formula is: emb=E clip (and) Among them, c is the concept embedding matrix, y is the target concept; Then, based on the knowledge distillation framework, the model is fine-tuned to make the target concept and the empty concept have the same characteristics in the text space. The calculation formula is: Among them, emb θ (c) Represent the text embedding of target concept c using the student CLIP model with parameters θ; and The parameters are θ * The frozen teacher CLIP model represents the text embeddings of the null concept c0 and the target concept c, and the coefficients a and β control the contribution of the embedding from the teacher model in the loss function.

7. The text-to-image diffusion fast concept erasure system according to claim 6, characterized in that: In the text-to-image diffusion module, the calibration set is introduced to design other concept retention strategies as follows: a concept retention loss function based on the difference between embeddings is designed, and the calculation formula is: where δ is the regularization coefficient used to balance the trade-off between retaining other concepts and erasing the target concept, c i denotes other concepts that are randomly selected from the calibration set and are different from the target concept c.

8. The text-to-image diffusion fast concept erasing system according to claim 6, characterized in that: In the generation module, the details are as follows: by inputting a text prompt, the model generates an image corresponding to the text description without containing the target concept.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is loaded into a processor, the method for rapid concept erasure from text to image diffusion according to any one of claims 1 to 4 is implemented.

10. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, a text-to-image diffusion fast concept erasure method according to any one of claims 1-4 is implemented.

Images