Generation graph diffusion model concept erasing method based on semantic enhancement and collaborative retention

Abstract

The invention provides a text graph diffusion model concept erasing method based on semantic enhancement and collaborative retention. The method comprises the steps that a target concept needing to be erased is defined, an erasure model is initialized based on a pre-trained diffusion model, preheating training is conducted on the initialized erasure model through the target concept, and a preheated erasure model is obtained; through word embedding in a predefined template library, generating an aggressive prompt capable of triggering a target concept by using the preheated erasing model; updating the preheated erasure model by utilizing aggressive prompt through a self-check and self-erasure mechanism in the erasure model to obtain an updated erasure model; and through global semantic relation alignment and local prediction noise retention, the generation capability of the updated erasing model on the non-target concept is optimized. According to the method, through a semantic enhancement erasing mechanism, erasing of the conceptual words is effectively expanded to the whole conceptual domain from a single vocabulary, and therefore the generalization ability of the model in different situations is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a concept erasing method of a cultural graph diffusion model based on semantic enhancement and collaborative preservation. Background Art

[0002] In order to alleviate the security risks that may be encountered while generating high-quality images, researchers have proposed a variety of methods, including pre-processing methods, post-processing methods, and model editing methods.

[0003] Pre-processing methods use pre-trained detectors to review and filter out images containing target content, and retrain the generated model with the filtered dataset, but the cost of retraining is high and may not completely filter out unwanted concepts. Post-processing methods use security checkers to identify inappropriate content and block risky outputs, but this method relies on the accuracy of the detector and may be limited by the inherent bias of the detector. Model editing methods erase or redirect target concepts by directly manipulating the original text diffusion model. This method is low-cost, flexible in operation, and has a significant erasure effect, so it is gaining more and more attention.

[0004] Model editing methods in the prior art are divided into four categories according to their technical characteristics: Guidance-based Methods, Attention Re-steering Methods, Closed-form Editing Methods, and Weight Masking Methods. These methods implement concept erasure from the perspectives of noise, image, text, and parameters, respectively. However, recent work has shown that existing concept erasure methods are not thorough enough. They use the idea of ​​text inversion to optimize prompts to stimulate the erased model to regenerate images of unsafe concepts. This shows that a more generalized concept domain erasure method is needed, rather than just modeling concepts as fixed vocabulary.

[0005] The disadvantages of the model editing methods in the above prior art include: when dealing with concept erasure, the existing methods often tightly bind concepts to specific words and perform erasure operations on these words. This method may cause the model to overfit specific words and ignore the fact that concepts may appear in multiple forms.

[0006] Existing concept erasure methods may not fully consider the possibility of adversarial attacks. Attackers can use adversarial prompts to test and challenge the erasure effect of the model by inducing the model to generate images of erased concepts through carefully constructed text prompts.

[0007] When performing concept erasure, existing techniques may affect the model's ability to understand and generate other non-target concepts. This is because the erasure process may not only simply suppress information related to a specific concept, but may also interfere with the model's overall understanding of language and visual features. Summary of the invention

[0008] The embodiment of the present invention provides a concept erasing method of a cultural image diffusion model based on semantic enhancement and collaborative preservation, so as to effectively perform target concept erasing on cultural images.

[0009] In order to achieve the above object, the present invention adopts the following technical scheme.

[0010] A concept erasing method for a cultural graph diffusion model based on semantic enhancement and synergy preservation, comprising:

[0011] Define a target concept to be erased, initialize an erasure model based on a pre-trained diffusion model, and perform preheating training on the initialized erasure model using the target concept to obtain a preheated erasure model;

[0012] Generate offensive prompts that can trigger target concepts using the pre-warmed erasure model through word embeddings in a pre-defined template library;

[0013] Using the aggressive prompt, the preheated erasure model is updated through a self-checking and self-erasing mechanism within the erasure model to obtain an updated erasure model;

[0014] The updated erasure model is optimized for its generative ability towards non-target concepts through global semantic relationship alignment and local prediction noise preservation.

[0015] Preferably, the target concept to be erased is defined, the erasure model is initialized based on the pre-trained diffusion model, and the initialized erasure model is preheated and trained using the target concept to obtain the preheated erasure model, including:

[0016] Initializing the current erasure model using a pre-trained diffusion model, wherein the erasure model and the diffusion model have the same structure and the same initial parameters, but the parameters of the erasure model will change as the training progresses;

[0017] Define the target concept to be erased, use the target concept to preheat the initial erasure model, and replace the prompt word τ representing the target concept cInput to the text encoder to guide erasure, the erasure formula is the same as formula (1) below;

[0018]

[0019] z t represents the latent variable with noise at time step t of the diffusion process, is the currently trained erasure model θ n The predicted noise, E is the statistical expectation, L′ erase To warm up the erasure loss in training;

[0020] A preheated erasure model is obtained, so that the erasure model can form a preliminary erasure capability for the target concept.

[0021] Preferably, the method of using the pre-heated erasure model to generate an offensive prompt capable of triggering a target concept through word embedding in a pre-defined template library includes:

[0022] The aggressive prompt is initialized by combining the target concept and a template randomly selected from a predefined template library, and the raw prompt of the aggressive prompt is expressed as τ p =[τ t ,τ c ], where τ t and τ c Represent the embeddings of template and target concept words respectively;

[0023] Introduce the aggregation function g(*) to summarize the text embedding and measure the original erasure model θ 0 And the erased model θ during training n The relationship between , using CLIP's native pre-trained aggregator as g(*), As the anchor feature of the concept domain, is the original model θ 0 The text encoder, τ P is the offensive cue that remains unchanged during the optimization process, and the original model θ is further trained in the target concept domain 0 , so that the original model θ 0 Ability to generate valuable concepts that are different from the concept anchor

[0024] Tip: Use standard H 1 To maintain As close as possible to the anchor feature Ensure that the target concept's offensive cues can reproduce the target concept content:

[0025]

[0026] Where Sim(*,*) is the cosine similarity metric function;

[0027] Use standard H 2 To ensure As far away from the anchor feature as possible

[0028]

[0029] By reducing H 2 And increase H 1 ,make sure Generated content with anchors As different as possible, by giving aggressive cues τ P Added adaptive noise implementation The generated content remains within the target concept domain, using the attack loss L attack To jointly optimize H 1 and H 2 :

[0030]

[0031] After the above optimization process, the aggressiveness prompt τ p As input for erasing model training.

[0032] Preferably, the method of updating the preheated erasure model by using the aggressive prompt through a self-checking and self-erasing mechanism within the erasure model to obtain an updated erasure model includes:

[0033] The target concept is converted into a self-enhanced tag embedding that is continuously updated based on training feedback. The self-enhanced tag embedding is used to explore the boundary of the target concept domain. The diffusion trajectory under the target concept related prompts is redirected by directing the guidance noise generated by the optimized self-enhanced tag embedding in the opposite direction to that before optimization:

[0034]

[0035] Where L erase is the erasure loss in the erasure model update process, z t represents the noisy latent variable at time step t of the diffusion process, τ P For offensive prompts, is the original diffusion model θ 0 The predicted noise, is the currently trained erasure model θ n The predicted noise, η is the erasure guidance parameter, is the negative guide noise, E is the statistical expectation;

[0036] DM uses the denoiser to predict the noise ∈ θ, before performing semantic enhancement erasure, the following formula is used for the warm-up training phase:

[0037]

[0038] So that θ n right There is an initial position, τ c It is the specified fixed prompt word.

[0039] Preferably, the optimization of the generation capability of the updated erasure model for non-target concepts by global semantic relationship alignment and local prediction noise preservation includes:

[0040] The global semantic consistency is maintained by constructing a semantic relationship graph of the retained hints, and the aggregation function g(*) is used to calculate the retained hint τ r In the original model θ 0 And the erased model θ during training n Semantic relationship graph M on :

[0041]

[0042] The consistency loss L of the semantic graph is calculated by the following formula graph :

[0043]

[0044] Introduced standard H 3 To measure θ 0 and θ n For the same τ r Semantic drift:

[0045]

[0046] According to H 3 The minimum value of the top-k tips is selected to form a subset C image , these hints will be used to constrain θ 0 and θ n Predicting noise consistency, achieving additional local noise consistency for subsets with weaker semantic alignment:

[0047]

[0048] Will erase the loss L erase , semantics Figure 1 Causative loss L graph and the noise prediction consistency loss L image Combined together, we form the overall loss L:

[0049] L(θ,θ 0 ,τ p,τ r )=L erase +γ t L graph +γ v L image (11)

[0050] Among them, γ t and γ v is the regularization parameter.

[0051] It can be seen from the technical solutions provided by the above embodiments of the present invention that the present invention effectively extends the erasure of concept words from a single word to the entire concept domain through the semantic enhancement erasure mechanism. This method is not only effective for text prompts directly related to the target concept, but also for cryptic or offensive prompts, thereby significantly improving the generalization ability of the model in different situations.

[0052] Additional aspects and advantages of the present invention will be given in part in the following description, which will become obvious from the following description, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0054] Figure 1 A schematic diagram of the implementation principle of a concept erasing method of a cultural graph diffusion model based on semantic enhancement and collaborative retention provided by an embodiment of the present invention;

[0055] Figure 2 A processing flow chart of a concept erasing method of a text graph diffusion model based on semantic enhancement and collaborative retention provided by an embodiment of the present invention;

[0056] Figure 3 A schematic diagram of a parameter updating process of an erasure model provided by an embodiment of the present invention;

[0057] Figure 4 A schematic diagram of an example of generating an image after deleting the concept of nudity provided by an embodiment of the present invention;

[0058] Figure 5 A schematic diagram of an example of generating an image after deleting the Van Gogh style provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0060] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or coupling. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items.

[0061] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0062] To facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention.

[0063] Concept erasure in the Vincent graph diffusion model guides conditional predictions away from the target concept by predicting negative guidance noise. In ESD (Erased Stable Diffusion), the diffusion process achieves concept erasure by adjusting the prediction noise so that the prediction noise is opposite to the generated noise under the target concept prompt condition. This method trains the model by minimizing the erasure loss, which is defined as the difference between the prediction noise and the reverse noise of the target concept. In the context of concept erasure, red team engineering evaluates the model's resistance to erasure mechanisms by generating aggressive prompts. The present invention actively generates aggressive prompts during model training and uses them to enhance the concept erasure effect, and improves the generalization of model erased concepts through iterative self-detection and self-erasure. At the same time, the present invention proposes a new aggressive prompt generation method for training, and a new irrelevant concept retention method.

[0064] The implementation principle diagram of a concept erasing method of a text graph diffusion model based on semantic enhancement and collaborative preservation provided by an embodiment of the present invention is as follows: Figure 1 As shown. The present invention uses two strategies to solve the concept erasure problem: semantic enhancement erasure and global-local collaborative retention mechanism. The present invention integrates three key components: offensive hint generation, semantic enhancement erasure and global-local collaborative retention. The offensive hint generation branch focuses on exploring and utilizing the potential expression of the target concept, the semantic enhancement erasure branch is responsible for realizing the self-inspection and self-purification of the concept domain, and the global-local collaborative retention branch ensures that the model maintains the understanding and generation capabilities of other concepts during the erasure process.

[0065] The processing flow chart of a concept erasing method of a text graph diffusion model based on semantic enhancement and collaborative preservation provided by an embodiment of the present invention is as follows: Figure 2 As shown, the process includes the following steps:

[0066] Step S1: Based on the pre-trained diffusion model, define the required erasure target concept and initialize the erasure model. Use the erasure target concept to preheat the initialized erasure model so that the erasure model can establish a preliminary erasure capability for the erasure target concept.

[0067] Step S2: Using the word embedding in the predefined template library, the preheated erasure model obtained in step S1 is used to generate attack prompts (Attack Prompt Generation) that can trigger the target concept for subsequent semantic enhancement erasure.

[0068] Step S3: Using the offensive prompts generated in step S2, through the self-checking and self-erasing mechanism within the erasure model, the preheated erasure model obtained in step S1 is used to enhance the erasure capability of the target concept domain (Semantic-Augment Erasing).

[0069] Step S4: By aligning global semantic relationships and preserving local prediction noise, the updated erasure model obtained in step S30 is optimized to generate non-target concepts, achieving effective erasure of target concepts and good retention of non-target concepts (Global-Local Collaborative Retention). The obtained erasure model is fine-tuned to further improve the erasure effect and image generation quality, ensuring the effectiveness and security of the erasure model in practical applications.

[0070] Specifically, the above step S1 includes: using a pre-trained diffusion model and a specified target concept, such as "nudity", to initialize the current erasure model. This step is the basis of the entire process, which ensures that the erasure model can recognize and prepare to erase specific target concepts. After initialization, the erasure model enters the warm-up training phase, which is trained using the target concept to enable the erasure model to form a preliminary erasure capability for the target concept, laying a solid foundation for subsequent offensive prompt generation and semantic enhancement erasure.

[0071] The erasure model is initialized from the original diffusion model, with the same structure and initial parameters. However, as the training progresses, the parameters of the erasure model will change, while the parameters of the original diffusion model remain unchanged.

[0072] Structural features of the diffusion model: It includes a text encoder from the CLIP CLIP (Contrastive Language-Image Pretraining) model, which is used to encode text features to guide the UNET denoiser to predict noise, a UNET network to generate prediction noise, and a variational autoencoder to encode and decode image features.

[0073] Warm-up process: using a prompt word representing the target concept, such as "naked", as the prompt word τ c , sent to the text encoder to guide erasure. The erasure formula is the same as formula (4) below.

[0074]

[0075] Specifically, the above step S2 includes: This step generates offensive prompts through an adversarial training process, and these offensive prompts can trigger the erasure model to generate images of the target concept. We first initialize the offensive prompts by combining the target concept and a template randomly selected from a predefined template library. For example, when erasing the concept "Van Gogh", we first initialize the offensive prompts with a template such as "An artwork by [Van Gogh]". For simplicity, the original prompt can be expressed as τ p =[τ t ,τ c ] where τ t and τ c Represent the embedding of template and concept words respectively.

[0076] To measure the original model θ 0 and the model θ in training n We introduce an aggregation function g(*) to summarize the text embeddings. Since the stable diffusion model uses the pre-trained CLIP text encoder, we use CLIP's native pre-trained aggregator as g(*). This allows us to obtain As the anchor feature of the concept domain, is the original model θ 0 The text encoder, τ P is the original τ that remains unchanged during the optimization process P .

[0077] Next, in order to make the current model θ n The key to escape from the word concept abyss is to further train θ in the target concept domain. 0 , which enables it to generate valuable concept hints that are different from concept anchors. We propose the standard H 1 To maintain As close as possible to the anchor feature Ensure that the target concept's offensive cues can reproduce the target concept content: Where Sim(*,*) is the cosine similarity metric function. At the same time, in order to encourage τ P To explore the boundaries of the concept domain, we further proposed the standard H 2 To ensure As far away from the anchor feature as possible By reducing H 2 And increase H 1 , we ensure Generated content with anchors as different as possible while still staying within the target concept domain. This is done by adding PThis is achieved by adding adaptive noise, which depends on and Finally, we propose the attack loss L attack To jointly optimize H 1 and H 2 :

[0078]

[0079] In this way, the aggressive cue τ p After a predefined optimization step, the traditional target concept τ is replaced c Become the input for erasure model training.

[0080] Specifically, the above step S3 includes: This step proposes a semantic enhancement erasure method, which allows the erasure model to explore and forget the boundary representation of the target concept domain through self-examination and self-erasure. Unlike the traditional fixed word concept representation, the present invention converts the concept words into self-enhanced tag embeddings that can be continuously updated based on training feedback. By freezing the denoiser responsible for image generation and adjusting the text encoder that manages the conditional mapping, we establish a connection between the text space of the current diffusion model and the visual space of the original diffusion model. In this way, the self-enhanced tag embedding can automatically explore the boundaries of the target concept domain by making the concept representation close to the target concept in the conditional text space and pushing it away from the target concept in the visual generation space.

[0081] To achieve this, the present invention proposes the following erasure loss L in the erasure model update process: erase , which redirects the diffusion trajectory under target concept-related cues by embedding the guidance noise generated by the optimized self-enhanced tag in the opposite direction to that before optimization:

[0082]

[0083] where DM uses the denoiser to predict the noise ∈ θ , and before semantic enhancement erasing, the following formula is used for the warm-up training phase:

[0084]

[0085] So that θ n right There is a relatively accurate initial positioning. Through this method, the present invention can effectively achieve the erasure of the target concept while maintaining the practicality and generation ability of the erasure model. This semantic enhancement erasure strategy not only improves the efficiency of the erasure model, but also enhances the generalization ability of the erasure model to the target concept domain through adaptive noise addition.

[0086] Specifically, the above step S4 includes: while implementing concept erasure, in order to maintain the availability of the erasure model, the present invention introduces a global-local collaborative retention mechanism. The core of this mechanism is to protect non-target concepts by maintaining the global consistency of semantic relationships and the stability of local noise prediction.

[0087] First, the present invention maintains global semantic consistency by constructing a semantic relationship graph of retention hints. Specifically, the aggregation function g(*) is used to calculate the retention hint τ r In the original model θ 0 and the model θ in training n Semantic relationship graph M on :

[0088]

[0089] Then, the consistency loss L of the semantic graph is calculated by the following formula graph :

[0090]

[0091] In order to further maintain the quality of image generation, the standard H 3 To measure θ 0 and θ n For the same τ r Semantic drift:

[0092]

[0093] According to H 3 The minimum value of the top-k tips is selected to form a subset C image These hints will be used to constrain θ 0 and θ n Predicting noise consistency, achieving additional local noise consistency for subsets with weaker semantic alignment:

[0094]

[0095] Finally, we will erase the loss L erase , semantics Figure 1 Causative loss L graph and the noise prediction consistency loss L image Combined together, we form the overall loss L:

[0096] L(θ,θ 0 ,τ p ,τ r )=L erase +γ t L graph +γ v L image(11)

[0097] Among them, γ t and γ v is the regularization parameter.

[0098] Through this method, the present invention ensures that the generation capability and semantic consistency of the erasure model on other concepts are protected while erasing the target concept, thereby achieving effective global-local collaborative preservation.

[0099] In the model fine-tuning step, the final erasure model is fine-tuned to further improve the erasure effect and image generation quality, ensuring the effectiveness and security of the model in practical applications. The input of the whole process includes the pre-trained diffusion model, target concept, template library and training data, and the output is a fine-tuned diffusion model with efficient concept erasure capability and useful knowledge retention. Figure 3 As shown, the processing steps include the following:

[0100] Original word: τ t and τ c Represent the embedding of template and concept word respectively, and the original prompt can be expressed as τ p =τ c .

[0101] Get the attack word: set the offensive prompt τ p =[τ t ,τ c ] and calculate the anchor feature

[0102] Calculate the attack loss: Calculate and Ensure that the target concept's offensive cues can reproduce the target concept content: To encourage τ P To explore the boundaries of the concept domain, we further calculate the standard Final calculation

[0103] Update attack template: rely on the calculated L attack To update the aggressiveness prompt τ p .

[0104] Computational semantics Figure 1 Consistency loss: Calculate the semantic relationship graph M(θ 0 ,τ r ) and M(θ i ,τ r ), so the semantics is calculated by formula (7) Figure 1 Causative loss graph .

[0105] Calculate the noise prediction loss: Choose H 3 The top-k tips corresponding to the minimum value are retained as the image set C image , the noise prediction consistency loss L is calculated by formula (9) image .

[0106] Calculate the erasure loss: Formula (2) can be used to obtain Therefore, the erasure loss can be calculated by formula (3).

[0107] Update DM parameters: After calculating the above three losses, we can calculate the total loss through formula (10), and update the erasure model parameters θ accordingly n .

[0108] Validation of the concept of erasing nudity: Figure 4 A schematic diagram of an example of generating an image after deleting the concept of nudity provided by an embodiment of the present invention. Figure 4 Experimental results are presented to visually compare the effects of different methods on the nudity concept erasure task. The present invention is shown to reduce explicit (nudity) content while maintaining high image quality and semantic consistency, and it better preserves the practicality of the model while erasing the target concept compared to other methods. In addition, the present invention shows strong generalization ability when facing offensive cues, which indicates that it is more powerful in practical applications.

[0109] Table 1 Quantitative evaluation of removing explicit content

[0110]

[0111] According to the quantitative data in Table 1, after using the present invention, the number of explicit content detected by the nudity detector (NudeNet) on the inappropriate image prompt (I2P) dataset is significantly reduced. Specifically, compared with other methods, the present invention greatly reduces the detected nude content while maintaining a high image generation quality while significantly reducing the exposure ratio (Relative Exposure Ratio, RER). The low score on the Fréchet Inception Distance (FID) indicator means that the distribution difference between the generated image and the original dataset is small, indicating that the quality and diversity of the image are better preserved during the erasing process. At the same time, SAGE scores higher on the CLIP-score, indicating that the generated image has a stronger semantic consistency with the text prompt. This shows the effectiveness of the present invention in preventing the generation of revealing content.

[0112] Verification of the effectiveness of erasing artistic style: According to the quantitative results in Table 2, the present invention has a high erasure success rate (Ae) in erasing Van Gogh's artistic style, which means that the model can be effectively prevented from generating images in the style of Van Gogh. At the same time, the present invention also has a high retention success rate (Ar), which indicates that erasing a specific style will not affect the model's generation of images in the style of other artists. In addition, through the harmonic mean (Ha) indicator, the present invention demonstrates overall superior performance in erasing Van Gogh's style. The Ha indicator comprehensively considers the erasure success rate and the retention success rate, and a high Ha value indicates a high degree of effectiveness and accuracy in the task of erasing a specific style.

[0113] Table 2 Quantitative evaluation of erasing Van Gogh's artistic style

[0114]

[0115] Figure 5 This is a schematic diagram of an example of an image generated after deleting the Van Gogh style provided by an embodiment of the present invention. Figure 5 As shown, the images generated by the present invention not only successfully avoid the elements of Van Gogh's style, but also maintain a high level of visual quality, which is consistent with the lower FID value, indicating that the generated images have a high visual similarity with the original images.

[0116] Table 3 shows the time required to generate aggressive prompts using the present invention under different conditions. By comparing the "PerIteration" and "Prompt Generation" columns, we can see that the present invention has a significant advantage in efficiency. Compared with the AdvUnlearn method, our method is 7.5 times faster in each iteration and 62.1 times faster in a single aggressive prompt generation.

[0117] Table 3. Efficiency comparison between AdvUnlearn and SAGE.

[0118]

[0119] 4. Robust anti-attack

[0120] Table 4 shows the performance of the present invention in the face of different sets of aggressive prompts. By using the NudeNet detector, we perform well in reducing the amount of detected explicit content, reducing the detected explicit content by 95.6% on average, which shows that the present invention is very robust against aggressive prompts. The robustness of the present invention is also reflected in its ability to handle a variety of different types of aggressive prompts, including those generated by P4D and Ring-A-Bell. These prompt sets are designed to test the security of the Vincent graph diffusion model with erased concepts, and its performance in these tests proves that it can resist potential adversarial attacks in practical applications.

[0121] Table 4 Comparison of erasure effects under aggressive cues.

[0122]

[0123] In summary, compared with the existing methods, the embodiments of the present invention effectively extend the erasure of concept words from a single word to the entire concept domain through a semantically enhanced erasure mechanism. This method is not only effective for text prompts directly related to the target concept, but also for implicit or offensive prompts, thereby significantly improving the generalization ability of the model in different situations.

[0124] This paper introduces a novel global-local collaborative preservation mechanism, which maintains the consistency between concepts at the global level through the semantic relationship graph and maintains the image quality at the local level by predicting the consistency of noise. This method effectively protects the generation ability of non-target concepts and reduces the damage to the practicality of the model during the erasing process.

[0125] Compared with previous aggressive prompt generation methods that require a lot of computing resources for retraining or complex optimization techniques, the present invention demonstrates a more computationally efficient inside-out exploration aggressive prompt generation strategy, which accelerates adversarial training in terms of rapid generation of aggressive prompts, thereby significantly reducing computing time and resource consumption.

[0126] The present invention is not only effective in erasing exposed content and artistic style in response to the two major issues of unsafe generation and copyright disputes in the generative model, but also outstanding in maintaining image quality and semantic alignment. In addition, the present invention has also proved its reliability in practical applications in the robustness test when facing offensive prompts.

[0127] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0128] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention or certain parts of the embodiments.

[0129] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0130] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A concept erasing method for the cultural graph diffusion model based on semantic enhancement and collaborative preservation, characterized in that: include: Define a target concept to be erased, initialize an erasure model based on a pre-trained diffusion model, and perform preheating training on the initialized erasure model using the target concept to obtain a preheated erasure model; Generate offensive prompts that can trigger target concepts using the pre-warmed erasure model through word embeddings in a pre-defined template library; Using the aggressive prompt, the preheated erasure model is updated through a self-checking and self-erasing mechanism within the erasure model to obtain an updated erasure model; The updated erasure model is optimized for its generative ability towards non-target concepts through global semantic relationship alignment and local prediction noise preservation.

2. The method according to claim 1, characterized in that The target concept to be erased is defined, the erasure model is initialized based on the pre-trained diffusion model, and the initialized erasure model is preheated and trained using the target concept to obtain the preheated erasure model, including: Initializing the current erasure model using a pre-trained diffusion model, wherein the erasure model and the diffusion model have the same structure and the same initial parameters, but the parameters of the erasure model will change as the training progresses; Define the target concept to be erased, use the target concept to preheat the initial erasure model, and replace the prompt word τ representing the target concept c Input to the text encoder to guide erasure, the erasure formula is the same as formula (1) below; z t represents the noisy latent variable at time step t of the diffusion process, is the currently trained erasure model θ n The predicted noise, E is the statistical expectation, L′ erase To warm up the erasure loss in training; A preheated erasure model is obtained, so that the erasure model can form a preliminary erasure capability for the target concept.

3. The method according to claim 2, characterized in that The method of using the pre-heated erasure model to generate offensive prompts that can trigger the target concept through word embedding in the pre-defined template library includes: The aggressive prompt is initialized by combining the target concept and a template randomly selected from a predefined template library, and the raw prompt of the aggressive prompt is expressed as τ p =[τ t ,τ c ], where τ t and τ c Represent the embeddings of template and target concept words respectively; The aggregation function g(*) is introduced to summarize the text embedding and measure the original erasure model θ0 and the erasure model θ in training. n The relationship between , using CLIP's native pre-trained aggregator as g(*), As the anchor feature of the concept domain, is the text encoder of the original model θ0, τ P It is the offensive prompt that remains unchanged during the optimization process. The original model θ0 is further trained in the target concept domain so that the original model θ0 can generate valuable concept prompts different from the concept anchor. The standard H1 is used to maintain As close as possible to the anchor feature Ensure that the target concept's offensive cues can reproduce the target concept content: Where Sim(*,*) is the cosine similarity metric function; Use standard H2 to ensure As far away from the anchor feature as possible By reducing H2 and increasing H1, ensure Generated content with anchors As different as possible, by giving aggressive cues τ P Added adaptive noise implementation The generated content remains within the target concept domain, using the attack loss L attack To jointly optimize H1 and H2: After the above optimization process, the aggressiveness prompt τ p As input for erasing model training.

4. The method according to claim 3, characterized in that The method of updating the preheated erasure model by using the aggressive prompt through the self-checking and self-erasure mechanism inside the erasure model to obtain the updated erasure model includes: The target concept is converted into a self-enhanced tag embedding that is continuously updated based on training feedback. The self-enhanced tag embedding is used to explore the boundary of the target concept domain. The diffusion trajectory under the target concept related prompts is redirected by directing the guidance noise generated by the optimized self-enhanced tag embedding in the opposite direction to that before optimization: Where L erase is the erasure loss in the erasure model update process, z t represents the noisy latent variable at time step t of the diffusion process, τ P For offensive prompts, is the noise predicted by the original diffusion model θ0, is the currently trained erasure model θ n The predicted noise, η is the erasure guidance parameter, is the negative guide noise, E is the statistical expectation; DM uses the denoiser to predict the noise ∈ θ , before performing semantic enhancement erasure, the following formula is used for the warm-up training phase: So that θ n right There is an initial position, τ c It is the specified fixed prompt word.

5. The method according to claim 4, characterized in that The method of optimizing the generation capability of the updated erasure model for non-target concepts by aligning global semantic relations and preserving local prediction noise includes: The global semantic consistency is maintained by constructing a semantic relationship graph of the retained hints, and the aggregation function g(*) is used to calculate the retained hint τ r The original model θ0 and the erased model θ during training n Semantic relationship graph M on : The consistency loss L of the semantic graph is calculated by the following formula graph : The H3 standard is introduced to measure θ0 and θ n For the same τ r Semantic drift: According to the minimum value of H3, select the top-k tips to form subset C image , these hints will be used to constrain θ0 and θ n Predicting noise consistency, achieving additional local noise consistency for subsets with weaker semantic alignment: Will erase the loss L erase , semantic graph consistency loss L graph and the noise prediction consistency loss L image Combined together, we form the overall loss L: L(θ,θ0,τ p ,t r )=L erase +g t L graph +g v L image (11) Among them, γ t and γ v is the regularization parameter.

Images