Text-to-image diffusion model concept erasing method, system, equipment and medium

Abstract

The invention discloses a text-to-image diffusion model concept erasing method, system, equipment and medium, which are corresponding schemes, in the scheme, from the perspective of concept generation probability, the generation probability of a target concept is integrally reduced, the forgetting of the target concept is really realized, and meanwhile, in order to avoid the influence caused by unrelated concepts in an image, the text-to-image diffusion model concept is erased. A concept localization method is provided, a target concept segmentation mask is introduced, introduction of irrelevant visual information is prevented, and accuracy and high efficiency of concept erasure are guaranteed; moreover, starting from the mechanism of the model, the generation of the target concept is further destroyed from the model level by disturbing the self-attention response of the model, so that the robustness of concept erasure is improved. Generally speaking, the method provided by the invention realizes robust and efficient concept erasure in a real sense from both a probability distribution level and a model level.

Description

Technical Field

[0001] The present invention relates to the technical field of text-to-image diffusion model concept erasure, and in particular to a text-to-image diffusion model concept erasure method, system, device and medium. Background Art

[0002] In recent years, generative models, especially text-to-image diffusion models, have developed rapidly. Due to their ability to generate high-quality images, such models have been widely used in many fields such as art creation, content generation, and advertising design. These models can generate high-quality and detailed images through natural language prompts.

[0003] However, since these models are usually trained based on large-scale data sets, the quality and source of the training data are often difficult to control, resulting in the model generation process possibly containing bad or sensitive content that is not suitable for public dissemination (hereinafter collectively referred to as bad content).

[0004] There are some existing methods that achieve the prevention and control of bad content by filtering training data. However, the filtering of bad data is limited, especially when faced with large-scale unprocessed network data, it is difficult to ensure that all bad content is eliminated. In addition, data filtering methods cannot effectively deal with the deep visual concepts that the model has learned. Therefore, some researchers have proposed using simpler and more efficient concept erasure technology to achieve the erasure or editing of learned concepts by fine-tuning the pre-trained model. However, existing concept erasure methods often rely on specific prompt words and can only erase the visual content corresponding to specific prompt words, but do not really erase the visual concepts from the model. If users use adversarial or paraphrased prompt words, they may still induce the model to generate bad content.

[0005] In view of this, the present invention is proposed. Summary of the invention

[0006] The purpose of the present invention is to provide a text-to-image diffusion model concept erasure method, system, device and medium, which can erase the target concept from the model by fine-tuning the model to obtain a safer and more reliable diffusion model.

[0007] The objective of the present invention is achieved through the following technical solutions: A text-to-image diffusion model concept erasing method, comprising: Get a given text-to-image diffusion model and a target concept image set; The target concept image set is subjected to noise processing and input into the text-to-image diffusion model; the target function is obtained by adjusting the target concept generation probability, and the corresponding target concept image is localized by generating a target concept segmentation mask, and a concept localization erasure loss function is constructed in combination with the target function; a self-attention distortion loss function is constructed in combination with the self-attention map generated by the text-to-image diffusion model to adjust the self-attention response; the text-to-image diffusion model is fine-tuned and trained in combination with the concept localization erasure loss function and the self-attention distortion loss function to obtain a text-to-image diffusion model with the target concept erased.

[0008] A text-to-image diffusion model concept erasure system, used to implement the aforementioned method, the system comprising: A model and image set acquisition unit, used to acquire a given text-to-image diffusion model and a target concept image set; A fine-tuning training unit is used to perform noise processing on the target concept image set and input it into the text-to-image diffusion model; obtain the target function by adjusting the target concept generation probability, and localize the corresponding target concept image by generating a target concept segmentation mask, and construct a concept localization erasure loss function in combination with the target function; construct a self-attention distortion loss function in combination with the self-attention map generated by the text-to-image diffusion model to adjust the self-attention response; fine-tune the text-to-image diffusion model in combination with the concept localization erasure loss function and the self-attention distortion loss function to obtain a text-to-image diffusion model with the target concept erased.

[0009] A processing device, comprising: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0010] A readable storage medium stores a computer program, which implements the above method when the computer program is executed by a processor.

[0011] It can be seen from the technical solution provided by the present invention that, on the one hand, from the perspective of concept generation probability, the generation probability of the target concept is reduced holistically, and the target concept is truly forgotten. At the same time, in order to avoid the influence of irrelevant concepts in the image, a concept localization method is proposed, and a target concept segmentation mask is introduced to prevent the introduction of irrelevant visual information, thereby ensuring the accuracy and efficiency of concept erasure. On the other hand, starting from the mechanism of the model itself, by disrupting the self-attention response of the model, the generation of the target concept is further destroyed from the model level, thereby improving the robustness of concept erasure. In general, the present invention takes both the probability distribution level and the model level into consideration, and truly realizes robust and efficient concept erasure. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0013] Figure 1 A flowchart of a text-to-image diffusion model concept erasing method provided by an embodiment of the present invention; Figure 2 A schematic diagram of the overall framework of a text-to-image diffusion model concept erasing method provided by an embodiment of the present invention; Figure 3 A schematic diagram showing a visual comparison of the erasing effects of the method of the present invention and the existing conceptual erasing method provided in an embodiment of the present invention; Figure 4 A schematic diagram of a text-to-image diffusion model concept erasure system provided by an embodiment of the present invention; Figure 5 A schematic diagram of a processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0014] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0015] First, the terms that may be used in this article are explained as follows: The terms "include", "comprises", "contains", "has" or other descriptions with similar semantics should be interpreted as non-exclusive inclusion. For example, including certain technical feature elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.) should be interpreted as including not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.

[0016] The term "consisting of..." means excluding any technical feature elements not explicitly listed. If this term is used in a claim, it will make the claim closed, so that it does not contain technical feature elements other than the technical feature elements explicitly listed, except for the conventional impurities related to them. If this term only appears in a clause of a claim, it only limits the elements explicitly listed in the clause, and the elements recorded in other clauses are not excluded from the overall claim.

[0017] The following is a detailed description of a text-to-image diffusion model concept erasing method, system, device and medium provided by the present invention. The contents not described in detail in the embodiments of the present invention belong to the prior art known to professional and technical personnel in the field. If no specific conditions are specified in the embodiments of the present invention, the conventional conditions in the field or the conditions recommended by the manufacturer are followed. The instruments used in the embodiments of the present invention, if the manufacturer is not specified, are all conventional products that can be purchased commercially.

[0018] Embodiment 1 The present invention provides a text-to-image diffusion model concept erasing method, such as Figure 1 As shown, it mainly includes the following steps: Step 1: Get the given text-to-image diffusion model and the target concept image set.

[0019] In an embodiment of the present invention, the given text-to-image diffusion model is an existing model, specifically a text-to-image diffusion model that requires erasing a target concept, and the target concept image set also belongs to an existing image data set. The above two types of information (i.e., the model and the image set) can be obtained by conventional means.

[0020] Step 2: Construct the loss function from the probability distribution level and model level, and perform fine-tuning training.

[0021] In an embodiment of the present invention, the target concept image set is subjected to noise processing and input into a text-to-image diffusion model; an objective function is obtained by adjusting the target concept generation probability, and the corresponding target concept image is localized by generating a target concept segmentation mask, and a concept localization erasure loss function is constructed in combination with the objective function; a self-attention distortion loss function is constructed in combination with the self-attention map generated by the text-to-image diffusion model to adjust the self-attention response; the text-to-image diffusion model is fine-tuned and trained in combination with the concept localization erasure loss function and the self-attention distortion loss function to obtain a text-to-image diffusion model with the target concept erased.

[0022] The preferred implementation of this step is as follows: (1) Construct a concept localization erasure loss function.

[0023] (1.1) The objective function is obtained by adjusting the target concept generation probability.

[0024] Set of target concept images Recorded as ,in, is a single target concept image, i=1,2,…,n, n is the number of target concept images; after the target concept image set is subjected to noise processing, a single target concept image after noise processing is recorded as , based on the variational lower bound, we get: ; in, Represents the target concept image at time step t Adding Noise The mathematical expectation after The noise predicted by the text-to-image diffusion model (referred to as predicted noise), is proportional to the sign, exp is an exponential function with the natural constant e as the base, Text-to-image diffusion model The probability of generating the target concept, is the noise, and t is the time step of the noise adding process.

[0025] By minimizing the probability of generating the target concept , determine the objective function as: ; in, is the objective function.

[0026] (1.2) Localizing the corresponding target concept image by generating a target concept segmentation mask, the steps are as follows: introducing a segmentation model; for each target concept image, segmenting it using the segmentation model to obtain a target concept segmentation mask; using the target concept segmentation mask, eliminating visual information irrelevant to the target concept.

[0027] (1.3) Construct the concept localization erasure loss function.

[0028] The concept of localized erasure loss function is expressed as: ; in, is the concept localization erasure loss function, M is the target concept segmentation mask, It is the Hadamard product symbol, which represents the multiplication of corresponding elements of vectors.

[0029] (2) Construct a self-attention distortion loss function.

[0030] (2.1) Aggregate all self-attention maps and select the vector corresponding to the largest singular value through singular value decomposition.

[0031] Specifically, the number of attention maps is recorded as K, and the dimension of each self-attention map is set for upsampling, and then aggregated to obtain the aggregated self-attention map ; Through singular value decomposition, we get: ; Among them, U and V are unitary matrices obtained by decomposition; S is a diagonal matrix, and SVD represents singular value decomposition. for The mean matrix of Vector, that is, the diagonal elements in the diagonal matrix are singular values, and the row with the largest singular value is denoted as Y. The row vector of the Yth row in the unitary matrix V is vector.

[0032] (2.2) Using the vector corresponding to the maximum singular value, construct the self-attention distortion loss function.

[0033] (3) Construct the final loss function and perform fine-tuning training.

[0034] (3.1) Combining the concept localization erasure loss function and the self-attention distortion loss function, the final loss function L is constructed, which is expressed as: ; in, , The corresponding representation concept localization erasure loss function and self-attention distortion loss function are and are two weight coefficients.

[0035] (3.2) Use the final loss function L to fine-tune the text-to-image diffusion model.

[0036] The present invention can erase the target concept (that is, the specific concept that needs to be erased) in the text-to-image diffusion model, so as to be used in scenarios such as privacy protection and model security control. In practice, it can be installed in a computer, mobile phone or other device in the form of software to provide real-time target concept erasure; it can also be installed in a server to provide large-scale target concept erasure.

[0037] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the method provided by the embodiment of the present invention is described in detail with specific embodiments below.

[0038] 1. Overall overview of the plan.

[0039] The embodiment of the present invention provides a solution that can erase high-probability generation areas in the text-to-image diffusion model. Unlike the existing erasure method based on prompt words, the present invention does not rely on specific prompt words for optimization, but considers both the probability distribution level and the model level to achieve the erasure effect. The solution of the present invention includes: generation probability adjustment, concept localization, and self-attention distortion. The generation probability adjustment derives the objective function through the variational lower bound (ELBO) to maximize the generation error of the model for the target concept. Concept localization localizes the concept image by generating a target concept segmentation mask, so that the model only focuses on the target concept area during the fine-tuning process. Self-attention distortion destroys the model's ability to understand the structure and shape of the target concept by disrupting the model's self-attention response. Thanks to the above improvements, the present invention significantly improves the performance of concept erasure and reaches an advanced level on the evaluation data set.

[0040] 2. Detailed introduction of the plan.

[0041] like Figure 2 As shown, the solution of the present invention is mainly divided into generation probability adjustment, concept localization, and self-attention distortion. The above parts are introduced respectively below.

[0042] 1. The generation probability has been adjusted.

[0043] Given a set of target concept images and pre-trained text-to-image diffusion model First, each target concept image is denoised to obtain a noisy image . From the derivation of the variational lower bound (ELBO), we have the following conclusions: ; in, Represents the target concept image at time step t Adding Noise The mathematical expectation after is proportional to the sign, exp is an exponential function with the natural constant e as the base, For Model The probability of generating the target visual concept, is the added noise, and t is the time step of adding noise. Therefore, the optimization goal of concept erasing can be defined as minimizing the generation probability of the target visual concept, that is: ; Derived from the above variational lower bound, the objective function can be defined as: .

[0044] The above objective function defines concept erasure from the perspective of probability distribution, avoids the introduction of specific prompt words, and realizes true visual concept erasure.

[0045] 2. Localization of concepts.

[0046] In order to achieve more accurate concept erasure and avoid the influence of irrelevant concept information in the image, the present invention proposes concept localization. Given any image, the Segment Everything Model (SAM) can segment different instances. Therefore, the target concept image is segmented using SAM, and the mask M (target concept segmentation mask) of the target concept is selected. In the concept erasure process, visual information irrelevant to the target concept is omitted. Combined with the above objective function, the concept localization erasure loss function can be defined as: .

[0047] 3. Self-attention distortion.

[0048] In order to further erase the target concept from the model level, the present invention proposes self-attention distortion. In the text-to-image diffusion model, the self-attention mechanism reflects the model's understanding of the shape, structure and other information of the visual concept, and the concept erasure effect at the model level can be achieved by destroying the model's self-attention.

[0049] The diffusion model generates K self-attention maps with different resolutions. For example, the U-Net in the text-to-image diffusion model generates 16 self-attention maps with different resolutions, such as 64×64×64×64, 32×32×32×32, 16×16×16×16, and 8×8×8×8.

[0050] In the embodiment of the present invention, the aggregation result is obtained by upsampling the set dimensions (e.g., the last two dimensions) of all self-attention maps to a specified resolution (e.g., the highest resolution) and then performing aggregation. .

[0051] For Self-attention map The processing is expressed as: ; Among them, Upsampling is an upsampling operation, for example, it can be implemented by a bilinear interpolation operation Bilinear. For the Self-attention map The result after upsampling operation; is the symbol of a real number set, h and w are the height and width of the self-attention map, and 64×64 is an example of the resolution.

[0052] The aggregation process is expressed as: ; in, represents the first two dimensions, , represents the reference position when calculating the self-attention map, and the subsequent colon means taking all elements. The above formula shows The solution process of each value of the first two dimensions is the aggregation result after integration ; for , W is an element in the highest resolution, for example, W=64.

[0053] right Perform singular value decomposition to obtain: ; in, for The mean matrix of , U and V are the unitary matrices obtained by decomposition; S is a diagonal matrix, the diagonal elements in the diagonal matrix are singular values, and the row with the largest singular value is recorded as Y. The row vector of the Yth row in V is taken as , introduce the self-attention distortion loss function as follows: ; Among them, for Through the self-attention distortion loss function, the erasing effect of visual concepts can be achieved from the model level, making the model forget the shape, structure and other information of the target concept.

[0054] Combining the above three parts, the final loss function of the fine-tuning model during concept erasure is written as: ; Based on the above loss function L, the model is fine-tuned and trained; taking Stable Diffusion v1.4 (abbreviated as SDv1.4) as the benchmark model, three to five collected target concept images are used as the fine-tuning dataset (i.e., the target concept image set). During the fine-tuning process, Set to 1, Set to 500, use Adam (Adaptive Moment Estimation) optimizer, set learning rate to 0.0001, and set the total number of fine-tuning rounds to 10. Use CLIP score as the evaluation index of concept erasure effect, the smaller the value, the better the erasure effect. Considering that the relevant fine-tuning training process can be achieved by conventional means, it will not be described in detail.

[0055] Among them, Stable Diffusion v1.4 is the 1.4 version of the stable diffusion model, and CLIP is contrastive language-image pre-training, which is a multimodal model. Both are conventional models.

[0056] 3. Effect description.

[0057] Existing concept erasing methods are often based on specific prompt words. For example, to erase the visual concept of "Mickey Mouse", existing methods can only prevent the generation of the concept when the user gives the prompt word "Mickey Mouse". When the user gives the prompt word as a paraphrase, such as "Disney classic characters", the visual concept "Mickey Mouse" will still be generated, causing the erasure method to fail. This is because the existing methods only change the mapping relationship from text to image in the generative model, but do not actually erase the model's ability to generate the target visual concept. Therefore, existing concept erasing methods have poor robustness and can be easily bypassed by adversarial prompt words or paraphrased prompt words.

[0058] In view of the above problems, the above solution provided by the embodiment of the present invention defines the goal of concept erasure as minimizing the generation probability from the perspective of concept generation probability, provides a theoretical basis for the entire erasure process, is more explainable, and is based on the variational lower limit, derives a concept erasure loss function based on the predicted noise mean square error, which comprehensively reduces the generation probability of the target concept and truly achieves the forgetting of the target concept. In order to avoid the influence of irrelevant concepts in the image, a concept localization method is proposed. By introducing a mask of the target concept in the fine-tuning stage, the introduction of irrelevant visual information is prevented, thereby ensuring the accuracy and efficiency of concept erasure. In addition, starting from the mechanism of the diffusion model itself, the present invention points out the relationship between the self-attention layer and the model's visual understanding ability, and proposes a self-attention distortion loss function, which further destroys the generation of the target concept from the model level and improves the robustness of concept erasure.

[0059] Thanks to the above improvements, the present invention significantly improves the performance of concept erasing and reaches an advanced level on the evaluation dataset. Specifically: the evaluation dataset used selects and collects the paraphrase prompts corresponding to the three concepts of "Pikachu", "Mickey", and "Snoopy", and each concept has 15 paraphrase prompts. The CLIP score of the present invention on this dataset is 61.91, which is 1.59 higher than the best existing method UCE. Figure 3 As shown, a visual comparison of the erasing effect between the method of the present invention and the existing concept erasing method is provided, which shows that the method exhibits better robustness in dealing with paraphrase prompt words and has a more thorough erasing effect; Figure 3 In the above, ESD, FMN and UCE are all existing concept erasure schemes. ESD (Erased Stable Diffusion) is stable diffusion erasure, FMN (Forget-Me-Not) is a forgetting method for diffusion models, and UCE (Unified Concept Editing) is unified concept editing. It should be noted that the text-to-image diffusion model involved in the present invention can be applied to multiple languages. Figure 3 Only examples in English are provided. In actual applications, users can input text in the same language as needed.

[0060] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by means of software plus necessary general hardware platforms. Based on such understanding, the technical solutions of the above embodiments can be embodied in the form of software products, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0061] Embodiment 2 The present invention also provides a text-to-image diffusion model concept erasure system, which is mainly used to implement the method provided in the above embodiment, such as Figure 4 As shown, the system mainly includes: A model and image set acquisition unit, used to acquire a given text-to-image diffusion model and a target concept image set; A fine-tuning training unit is used to perform noise processing on the target concept image set and input it into the text-to-image diffusion model; obtain the target function by adjusting the target concept generation probability, and localize the corresponding target concept image by generating a target concept segmentation mask, and construct a concept localization erasure loss function in combination with the target function; construct a self-attention distortion loss function in combination with the self-attention map generated by the text-to-image diffusion model to adjust the self-attention response; fine-tune the text-to-image diffusion model in combination with the concept localization erasure loss function and the self-attention distortion loss function to obtain a text-to-image diffusion model with the target concept erased.

[0062] Considering that the processing details of each part involved in the system have been introduced in detail in the previous embodiments, they will not be repeated here.

[0063] Technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0064] Embodiment 3 The present invention also provides a processing device, such as Figure 5As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the methods provided in the aforementioned embodiments.

[0065] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0066] In the embodiment of the present invention, the specific types of the memory, input device and output device are not limited; for example: The input device may be a touch screen, an image acquisition device, a physical button or a mouse, etc.; The output device may be a display terminal; The memory may be a random access memory (RAM) or a non-volatile memory such as a disk storage.

[0067] Embodiment 4 The present invention also provides a readable storage medium storing a computer program, which implements the method provided in the above embodiment when the computer program is executed by a processor.

[0068] In the embodiment of the present invention, the readable storage medium as a computer-readable storage medium can be set in the aforementioned processing device, for example, as a memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk, etc., which can store program codes.

[0069] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or in any form that the information constitutes prior art known to those skilled in the art.

Claims

1. A text-to-image diffusion model concept erasing method, characterized in that: include: Get a given text-to-image diffusion model and a target concept image set; The target concept image set is subjected to noise processing and input into the text-to-image diffusion model; the target function is obtained by adjusting the target concept generation probability, and the corresponding target concept image is localized by generating a target concept segmentation mask, and a concept localization erasure loss function is constructed in combination with the target function; a self-attention distortion loss function is constructed in combination with the self-attention map generated by the text-to-image diffusion model to adjust the self-attention response; the text-to-image diffusion model is fine-tuned and trained in combination with the concept localization erasure loss function and the self-attention distortion loss function to obtain a text-to-image diffusion model with the target concept erased.

2. The text-to-image diffusion model concept erasing method according to claim 1, characterized in that: The objective function obtained by adjusting the target concept generation probability includes: Set of target concept images Recorded as ,in, is a single target concept image, i=1,2,…,n, n is the number of target concept images; After the target concept image set is subjected to noise processing, a single target concept image after noise processing is recorded as , based on the variational lower bound, we get: ； in, Represents the target concept image at time step t Adding Noise The mathematical expectation after is the noise predicted by the text-to-image diffusion model, is proportional to the sign, exp is an exponential function with the natural constant e as the base, Text-to-image diffusion model The probability of generating the target concept; By minimizing the probability of generating the target concept , determine the objective function as: ； in, is the objective function.

3. A text-to-image diffusion model concept erasure method according to claim 1 or 2, characterized in that: The localization processing of the corresponding target concept image by generating a target concept segmentation mask comprises: Introducing the Split Everything model; For each target concept image, the segmentation model is used to segment it and obtain the target concept segmentation mask; The target concept segmentation mask is used to remove visual information irrelevant to the target concept.

4. The text-to-image diffusion model concept erasing method according to claim 2, characterized in that: The concept localization erasure loss function is expressed as: ； in, is the concept localization erasure loss function, M is the target concept segmentation mask, is the Hadamard product symbol.

5. The text-to-image diffusion model concept erasing method according to claim 1, characterized in that: The self-attention map generated by the text-to-image diffusion model is combined to construct a self-attention distortion loss function, which includes: Aggregate all self-attention maps and select the vector corresponding to the maximum singular value through singular value decomposition; The self-attention distortion loss function is constructed using the vector corresponding to the largest singular value.

6. The text-to-image diffusion model concept erasing method according to claim 5, characterized in that: The method of aggregating all self-attention graphs and selecting the vector corresponding to the maximum singular value through singular value decomposition includes: The number of attention maps is recorded as K, and the dimension of each self-attention map is set for upsampling, and then aggregated to obtain the aggregated self-attention map ; Through singular value decomposition, we get: ； Among them, U and V are unitary matrices obtained by decomposition; S is a diagonal matrix, and SVD represents singular value decomposition. for The mean matrix of Vector, that is, the diagonal elements in the diagonal matrix are singular values, and the row with the largest singular value is denoted as Y. The row vector of the Yth row in the unitary matrix V is vector.

7. The text-to-image diffusion model concept erasing method according to claim 1, characterized in that: The fine-tuning training of the text-to-image diffusion model by combining the concept localization erasure loss function with the self-attention distortion loss function includes: Combining the concept localization erasure loss function and the self-attention distortion loss function, the final loss function L is constructed, which is expressed as: ； in, , The corresponding representation concept localization erasure loss function and self-attention distortion loss function are and are two weight coefficients; The text-to-image diffusion model is fine-tuned using the final loss function L.

8. A text-to-image diffusion model concept erasure system, characterized in that: For implementing the method described in any one of claims 1 to 7, the system comprises: A model and image set acquisition unit, used to acquire a given text-to-image diffusion model and a target concept image set; A fine-tuning training unit is used to perform noise processing on the target concept image set and input it into the text-to-image diffusion model; obtain the target function by adjusting the target concept generation probability, and localize the corresponding target concept image by generating a target concept segmentation mask, and construct a concept localization erasure loss function in combination with the target function; construct a self-attention distortion loss function in combination with the self-attention map generated by the text-to-image diffusion model to adjust the self-attention response; fine-tune the text-to-image diffusion model in combination with the concept localization erasure loss function and the self-attention distortion loss function to obtain a text-to-image diffusion model with the target concept erased.

9. A processing device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images