Sensitive content filtering and defending method for text generation graph model based on text embedding optimization

Abstract

The invention discloses a text embedding optimization-based sensitive content filtering and defending method for a text generation graph model, and relates to the technical field of large model security. The method comprises the steps of preparing a positive and negative sample data set and a common data set; parameters of a text encoder are adjusted, and a teacher-student model is used for training; stabilizing a text encoder by using a common data set; and designing a comprehensive loss function, performing training optimization in combination with the plurality of data sets to generate an adjusted student model, and generating a final security image through text embedding. According to the method, when text input containing sensitive vocabularies is processed through fine tuning of a text encoder, text embedding of the sensitive vocabularies is converted into text embedding of corresponding benign vocabularies, and the benign text embedding is used for guiding generation of a security image; on the premise of not changing the model generation capability, the generation of images containing sensitive pornography or other images protected by copyright by the text graph model is avoided, and the effective filtering, conversion and defense of sensitive contents are realized.

Description

Technical Field

[0001] The present invention belongs to the field of large model security technology, and in particular relates to a sensitive content filtering and defense method for a text-based graph model based on text embedding optimization. Background Art

[0002] In recent years, text-to-image models (T2I) have made significant progress, allowing users to generate synthetic images by inputting natural language descriptions (i.e., prompt words). Such models usually consist of a language model that understands the input prompt words and a component that generates images. For example, the text encoder of the Constrastive Language-Image Pre-training (CLIP) model or the Bidirectional Encoder Representations from Transformers (BERT) are used as language models, while the diffusion model is used for image generation. Taking the Stable Diffusion (SD) model as an example, its image generation process starts with a latent noise vector that converts the output of the text encoder into a latent image embedding. Subsequently, the image decoder of SD decodes the latent image embedding into an image. This approach combines the advantages of language models and image generation models, so that the generated image can highly fit the input text description.

[0003] Text-to-image models are usually trained on large image datasets. For example, the SD-1.5 model is trained on the Laion-5b dataset, which includes 5.8 billion images captured from the Internet. In such a large dataset, it is inevitable that sensitive, pornographic, or violent and bloody images will be included. Although the text-to-image model may be equipped with security filters, there is still a high probability that images containing inappropriate or sensitive content will be generated. This uncontrolled content generation will not only raise ethical issues, but may also bring legal risks and user experience issues.

[0004] Existing defense methods mainly include text hint filters, post-filters based on image detection, and fine-tuning of text graph models for concept erasure. Text hint filters block image generation by checking whether the input text hints contain any sensitive words before the image generation stage. Another defense strategy involves post-safety checkers, such as safety filters integrated into SD, which run at the output stage of the image generation process and check the generated images to ensure that they do not contain any "Not-Safe-For-Work" (NSFW) elements: used to mark those content that is not suitable for viewing at work or in public. In addition, recent defense methods mainly rely on concept erasure, which is fundamentally different from external safety measures because they modify the model's reasoning guidance or use fine-tuning to actively suppress NSFW concepts.

[0005] Text hint filters are vulnerable to adversarial examples. Adversarial examples can circumvent filters by modifying input text, making it difficult to detect and block sensitive content. Image-based filters such as CLIP have limited effectiveness in detecting sensitive content in generated images because CLIP is primarily trained for retrieval tasks rather than detail matching tasks. Methods that erase sensitive concepts may not completely eliminate NSFW concepts and may inadvertently affect the quality of benign images, and methods that fine-tune text-generated graph models require a lot of computing resources. Summary of the invention

[0006] The purpose of the present invention is to provide a sensitive content filtering and defense method for a text-based image model based on text embedding optimization, aiming to achieve effective filtering and conversion of sensitive content without changing the model generation capability, and prevent the generation of images of specific copyrighted content. This can be achieved through the following technical solutions:

[0007] The embodiment of the present application provides a sensitive content filtering and defense method of a text-based graph model based on text embedding optimization, comprising the following steps:

[0008] Prepare negative sample dataset T and normal dataset D;

[0009] Based on the negative sample dataset T, a positive sample dataset is generated using a large language model

[0010] Adjust the parameters of the text encoder and train it using the teacher-student model;

[0011] Stabilize the text encoder using a common dataset D;

[0012] Design a comprehensive loss function L, combine multiple data sets for training optimization and generate an adjusted student model;

[0013] Use the adjusted student model for text embedding to generate and output the final security image;

[0014] The teacher-student model includes a student model SE(·) and a teacher model TE(·).

[0015] Preferably, the negative sample data set T includes sensitive concepts and concepts of copyrighted objects; the general data set D includes commonly used text prompts; and the positive sample data set is the set of antonym sentences of negative samples in the negative sample dataset T.

[0016] Preferably, generate a positive sample dataset Specifically:

[0017] Given a negative sample t containing a violation description;

[0018] Use a large language model to generate positive samples corresponding to negative samples t

[0019] For concepts of copyrighted objects or other concepts to be deleted, the text embeddings of the terms are mapped to the embeddings of the placeholders.

[0020] Preferably, adjusting the parameters of the text encoder specifically includes the following contents:

[0021] Both the student model SE(·) and the teacher model TE(·) are initialized using the same pre-trained encoder weights as the original text-generated graph model;

[0022] Only the weights of the student model SE(·) are updated so that it maps the text embeddings of sensitive concepts to the corresponding security concepts;

[0023] Keep the parameters of the teacher model TE(·) unchanged.

[0024] Preferably, the teacher-student model is used for training, specifically:

[0025] Teacher model TE(·) processes positive samples And generate the corresponding positive sample text embedding

[0026] The student model SE(·) processes the negative sample t and generates the corresponding negative sample text embedding SE(t).

[0027] Preferably, a comprehensive loss function L is designed, including:

[0028] Calculate positive sample text embedding using cosine similarity The similarity between the negative sample text embedding SE(t) and the second loss function L2 is defined as:

[0029]

[0030] in, represents the negative sample text embedding SE(t) generated by the student model SE(·) and the positive sample text embedding generated by the teacher model TE(·) The cosine similarity of .

[0031] Preferably, it also includes:

[0032] Define the first loss function L1, expressed as:

[0033]

[0034] Among them, Cos(SE(d),TE(d)) represents the cosine similarity of the embedding vectors generated by the student model SE(·) and the teacher model TE(·) when processing the sampled common sample d on the common dataset D.

[0035] Preferably, it also includes:

[0036] Combine the first loss function L1 and the second loss function L2 and define the comprehensive loss function L, expressed as:

[0037] L = L1 + β·L2;

[0038] Among them, β is a weight coefficient used to balance the first loss function L1 and the second loss function L2.

[0039] Preferably, the final safety image is generated using the adjusted student model, expressed as:

[0040] img = G(SE(P));

[0041] Among them, SE() represents the adjusted student model and P represents the input text prompt.

[0042] The beneficial effects of the present invention are:

[0043] (1) The present invention introduces a method of fine-tuning a text encoder. When processing text input containing sensitive words, the text embeddings of these sensitive words are converted into text embeddings of corresponding benign words. The benign text embeddings are used to guide the image generation module to prevent the generation of dangerous pictures. Thus, by fine-tuning the text encoder, the text-image model is prevented from generating images containing sensitive, pornographic or other inappropriate content without changing the model generation capability, thereby achieving effective filtering and conversion of sensitive content and preventing and preventing the generation of copyrighted images.

[0044] (2) The present invention adjusts the parameters of the text encoder so that when processing text input containing sensitive words, it can map the embedding of these sensitive words to the embedding of non-sensitive benign words. Specifically, the text embedding vector of the original sensitive word is adjusted to be close to the text embedding vector of its corresponding benign word. For example, the text embedding of a sensitive word such as "naked" is mapped to the text embedding of "wearing clothes", thereby removing sensitive concepts in the text embedding stage and avoiding the generation of inappropriate images. This adjustment process ensures that the model can generate image content that meets ethical and legal requirements when processing sensitive content.

[0045] (3) The present invention uses a teacher-student model for training during the fine-tuning stage of the text encoder. The teacher model processes positive samples (such as "dressed woman"), while the student model processes negative samples (such as "naked woman"). By comparing the embedding vectors of positive and negative samples, the student model is optimized so that it generates an embedding vector similar to that of the positive sample when processing the negative sample. This ensures that even if the input text contains sensitive words, the generated image is appropriate.

[0046] (4) The present invention uses a common training set to stabilize the text encoder, ensuring that the overall performance of the text encoder will not deviate too much during the process of deleting sensitive concepts, thereby maintaining the generation capability and quality of the model. This can prevent the degradation of model performance due to sensitive word conversion.

[0047] (5) The present invention designs and optimizes a comprehensive loss function to combine the normal data set with the positive and negative sample data set for training. The comprehensive loss function improves the performance of the model when processing both positive and negative samples, thereby ensuring the security and appropriateness of the generated content. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] For better understanding and implementation, the technical solution of the present application is described in detail below with reference to the accompanying drawings.

[0049] Figure 1 A flowchart of a sensitive content filtering and defense method of a text-based graph model based on text embedding optimization provided in an embodiment of the present application;

[0050] Figure 2 A schematic diagram of a process for fine-tuning a text encoder provided in an embodiment of the present application;

[0051] Figure 3 A flow chart of the fine-tuned student model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to further explain the technical means and effects taken by the present invention to achieve the predetermined invention purpose, exemplary embodiments will be described in detail here, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of methods and systems consistent with some aspects of the present application as detailed in the attached claims.

[0053] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to any or all possible combinations of one or more associated listed items.

[0054] The specific implementation methods, features and effects of the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0055] Because a large amount of data captured from the Internet is usually used when training text-based graph models, but because the data set is not effectively filtered, there are many hidden dangers in the model's security; in addition, existing defense methods also have certain defects. Based on this, this application starts with the text encoder of the text-based graph model to make improvements: because images need to be generated based on the semantic information of the text, the most intuitive idea is that if the text encoder does not recognize sensitive words such as "nudity" and "sex", or encodes these sensitive words as their antonyms, that is, positive words, it can fundamentally prevent the use of sensitive text embeddings to guide image generation in the denoising process. This method can effectively reduce the sensitive content in the generated image and improve the security of the model.

[0056] Example 1

[0057] See also Figure 1 The embodiment of the present application provides a sensitive content filtering and defense method of a text-based graph model based on text embedding optimization, comprising the following steps:

[0058] S1, prepare a negative sample dataset T and a normal dataset D for normal training;

[0059] S2, based on the negative sample dataset T, uses the large language model to generate a positive sample dataset

[0060] S3, adjust the parameters of the text encoder and train it using the teacher-student model;

[0061] S4, using a common dataset D to stabilize the text encoder;

[0062] S5, design a comprehensive loss function L, combine multiple data sets for training optimization and generate an adjusted student model;

[0063] S6, uses the adjusted student model to embed text, generate and output the final security image;

[0064] The teacher-student model includes a student model SE(·) and a teacher model TE(·).

[0065] Specifically, this application adjusts the parameters of the text encoder so that when processing text input containing sensitive words, it can map these sensitive word embeddings to non-sensitive benign word embeddings. Specifically, the text embedding vectors of the original sensitive words are adjusted to be close to the text embedding vectors of their corresponding benign words. For example, the text embedding of a sensitive word such as "naked" is mapped to the text embedding of "wearing clothes", thereby removing sensitive concepts in the text embedding stage and avoiding the generation of inappropriate images. This adjustment process ensures that the model can generate image content that meets ethical and legal requirements when processing sensitive content.

[0066] During the fine-tuning phase of the text encoder, this embodiment uses a teacher-student model for training. The teacher model processes positive samples (such as "dressed woman"), while the student model processes negative samples (such as "naked woman"). By comparing the embedding vectors of positive and negative samples, the student model is optimized so that it generates an embedding vector similar to that of the positive sample when processing the negative sample. This ensures that even if the input text contains sensitive words, the generated image is appropriate.

[0067] Use a common training set to stabilize the text encoder and ensure that the overall performance of the text encoder does not deviate too much during the process of removing sensitive concepts, thereby maintaining the generative capacity and quality of the model. This can prevent the performance of the model from degrading due to sensitive word conversion.

[0068] Design and optimize a comprehensive loss function to combine the normal dataset with the positive and negative sample datasets for training. The purpose of this comprehensive loss function is to improve the performance of the model when processing both positive and negative samples, thereby ensuring the safety and appropriateness of the generated content.

[0069] This application uses the above steps and methods to more efficiently, safely and reliably block and filter unsafe content in the text-generated image model. This method controls the compliance of the generated content from the root and avoids potential legal and ethical issues.

[0070] In one embodiment provided in the present application, the negative sample data set T includes sensitive concepts and concepts of copyrighted objects; the common data set D includes commonly used text prompts; the positive sample data set is the set of antonym sentences of negative samples in the negative sample dataset T.

[0071] In one embodiment provided in the present application, a positive sample data set is generated Specifically:

[0072] Given a negative sample t containing a violation description;

[0073] Use a large language model to generate positive samples corresponding to negative samples t

[0074] For concepts of copyrighted objects or other concepts to be deleted, the text embeddings of the terms are mapped to the embeddings of the placeholders.

[0075] Specifically, the negative sample dataset T includes sensitive concepts or concepts of copyrighted objects that need to be deleted (such as "nudity", "sex", "Van Gogh", etc.), which usually lead to unsafe image generation. It is usually a collection of antonyms of negative samples, for example, "a naked woman" corresponds to "a woman wearing clothes". Specifically, "a naked woman" will generate an unqualified picture, while "wearing clothes" happens to be its antonym, avoiding the generation of unqualified images.

[0076] In this embodiment, a positive data set can be generated through a large language model. Specifically, given a negative sample t containing a violation description, a large language model is used to generate its corresponding positive sample. For concepts that need copyright protection or other concepts that need to be deleted, the text embeddings of these words are mapped to the embeddings of placeholders. In this way, by removing these sensitive or protected concepts in the text encoding stage, it is possible to effectively avoid generating inappropriate or illegal image content.

[0077] In addition, a normal training set D containing common text prompts is used, such as sentences containing descriptions of daily life, such as "a person reading a book", to stabilize the text encoder and ensure that the text embeddings of other words do not deviate during the removal of sensitive concepts. This prevents the generated images from being semantically irrelevant to the input prompts and ensures the generation ability and accuracy of the model.

[0078] In an embodiment provided in the present application, adjusting the parameters of the text encoder specifically includes the following contents:

[0079] Both the student model SE(·) and the teacher model TE(·) are initialized using the same pre-trained encoder weights as the original text-generated graph model;

[0080] Only the weights of the student model SE(·) are updated so that it maps the text embeddings of sensitive concepts to the corresponding security concepts;

[0081] Keep the parameters of the teacher model TE(·) unchanged.

[0082] In an embodiment provided in the present application, training is performed using a teacher-student model, specifically:

[0083] Teacher model TE(·) processes positive samples And generate the corresponding positive sample text embedding

[0084] The student model SE(·) processes the negative sample t and generates the corresponding negative sample text embedding SE(t).

[0085] Specifically, in the text encoder fine-tuning stage, the present embodiment starts with two text encoders TE(·) and SE(·), both of which are initialized using the same pre-trained encoder weights as originally used by the text graph model. Fine-tuning with pre-trained weights is better than retraining a text encoder; fine-tuning pre-trained weights is better than retraining a completely new text encoder. Since retraining a new text encoder will almost certainly generate different embeddings that may not match the encoder currently used to guide image generation, the entire text graph model also needs to be retrained or at least adjusted to accommodate these new embeddings. Initializing and fine-tuning with pre-trained encoder weights can ensure the consistency and stability of the embeddings, thereby improving the performance of the text graph model and the quality of the generated images. This not only saves training time and computing resources, but also ensures that the generated images are highly relevant to the semantics of the input text prompts.

[0086] In the student-teacher model framework, in order to ensure that sensitive concepts in the generated text embeddings are mapped to corresponding security concepts, this embodiment only updates the weights of the student model SE(·) so that it maps the text embeddings of sensitive concepts to the corresponding security concepts, and keeps the parameters of the teacher model TE(·) unchanged. Specifically, the teacher model TE(·) processes positive samples. Generate corresponding text embedding The student model SE(·) processes the negative sample t and generates the corresponding text embedding SE(t). It minimizes the difference between the two embedding vectors through a suitable distance or similarity loss function, which can be expressed as

[0087] At the same time, in order to ensure that the quality and appearance of the generated images when using the text-generated image model (text-generated image model) are similar to those when using the original encoder without significant deviation, a common dataset D is also used to ensure the practicality of the student encoder. This method is formulated as a bi-objective optimization problem, which aims to balance the effectiveness of removing sensitive concepts and maintaining the stability of normal input image generation. This optimization strategy not only improves the robustness and security of the model, but also ensures high standards in visual quality and semantic consistency of the generated images.

[0088] In an embodiment provided in the present application, a comprehensive loss function L is designed, which specifically includes the following contents:

[0089] Calculate positive sample text embedding using cosine similarity The similarity between the negative sample text embedding SE(t) and the second loss function L2 is defined as:

[0090]

[0091] in, represents the negative sample text embedding SE(t) generated by the student model SE(·) and the positive sample text embedding generated by the teacher model TE(·) The cosine similarity of

[0092] Define the first loss function L1, expressed as:

[0093]

[0094] Where Cos(SE(d),TE(d) represents the cosine similarity of the embedding vectors generated by the student model SE(·) and the teacher model TE(·) when processing the sampled common sample d on the common dataset D;

[0095] Combine the first loss function L1 and the second loss function L2 and define the comprehensive loss function L, expressed as:

[0096] L = L1 + β·L2;

[0097] Among them, β is a weight coefficient used to balance the first loss function L1 and the second loss function L2.

[0098] Specifically, this embodiment designs and optimizes a comprehensive loss function, which combines the common data set D with the negative sample data set T and the positive sample data set Combined for training, the purpose of this comprehensive loss function is to improve the performance of the model when dealing with both positive and negative samples, thereby ensuring the safety and appropriateness of the generated content. The specific contents are as follows:

[0099] In order to prevent the generation of unsafe concept images, the student model SE(·) (also called the student encoder SE(·)) needs to map the unsafe concepts to the text embeddings of their corresponding safe concepts. To achieve this goal, this embodiment uses cosine similarity to calculate the similarity of two text embeddings (positive sample text embedding and negative sample text embedding) and defines the following loss function:

[0100]

[0101] in, Denotes the negative sample text embedding SE(t) generated by the student model and the positive sample text embedding generated by the teacher model The cosine similarity of .

[0102] At the same time, in order to ensure that the student encoder does not affect the embedding effect of other texts, the following loss function is defined:

[0103]

[0104] where Cos(SE(d),TE(d)) represents the cosine similarity of the embedding vectors generated by the student model SE(·) and the teacher model TE(·) when processing sample d on the normal dataset D. By minimizing L1, we can ensure that the embedding generated by the student model is close to the embedding of the teacher model, thereby preserving the general utility of the encoder and ensuring the semantic consistency of the generated image with the normal input prompt.

[0105] In general, this embodiment combines the above two loss functions to define a comprehensive loss function L:

[0106] L = L1 + β·L2;

[0107] Among them, β is a weight coefficient used to balance the first loss function L1 of retaining general utility and the first loss L2 of removing sensitive concepts.

[0108] In an embodiment provided in the present application, the final safety image is generated using the adjusted student model, which is expressed as:

[0109] img = G(SE(P));

[0110] Among them, SE() represents the adjusted student model and P represents the input text prompt.

[0111] Specifically, this embodiment uses the student model for text embedding, and the original text encoder can be directly replaced with the trained student model. This replacement process does not require additional overhead and can be plug-and-play. This embodiment uses the fine-tuned student model to generate a text embedding vector to guide the final image generation, which is expressed as:

[0112] img = G(SE(P));

[0113] Among them, SE() represents the fine-tuned text encoder (student model) and P represents the input text prompt.

[0114] It should be noted that, in order to distinguish the student model before and after fine-tuning, this application uses SE(·) to represent the student model before fine-tuning, and uses SE() to represent the student model after fine-tuning.

[0115] In summary, this application aims to achieve effective filtering and conversion of sensitive content without changing the model generation capability, and prevent the generation of images of specific copyrighted content. By introducing a method of fine-tuning the text encoder, this application converts the text embeddings of these sensitive words into text embeddings of corresponding benign words when processing text input containing sensitive words, and uses benign text embeddings to guide the image generation module to prevent the generation of dangerous images.

[0116] Compared with existing methods such as concept erasing, this application does not need to fine-tune the denoising parameters of the entire diffusion model, but only needs to fine-tune the text encoder. The advantage of this method is that the fine-tuning speed of the text encoder is much faster than the fine-tuning of the denoising network, and the required GPU resources are much less. Modifying the parameters of the denoising network may inadvertently have a significant impact on the quality of the generated image, while the present invention only fine-tunes the parameters of the text encoder, so it will not affect the quality of the generated image.

[0117] In addition, this application can flexibly respond to different types of sensitive content by dynamically adjusting text embedding. This method is not limited to a predefined list of sensitive words, but can also adapt to newly emerging sensitive words and adversarial samples by continuously updating training data.

[0118] On the other hand, compared with traditional text filters, when processing text input containing sensitive words, the present application avoids the problem of completely blocking text input by converting these sensitive words into embeddings of benign words. For example, for copyrighted content such as "Van Gogh-style starry sky", the text filter will directly block the output when it detects "Van Gogh", but the present invention will generate a picture of a star map without Van Gogh's style.

[0119] Example 2

[0120] See also Figure 2-Figure 3 , this application also provides another implementation method, the specific contents are as follows:

[0121] Take the SD (Stable Diffusion) model as an example. The SD model is the most commonly used open source image model. With its excellent image generation capabilities and rich plug-in support, it has been loved by a large number of users. The model denoises the image in the latent space and uses the cross-attention mechanism to guide the denoising process.

[0122] The text prompt (or text input) P is processed by the text encoder of the CLIP model to generate a text embedding, which can be represented as E(P), and then sent to the cross-attention layer in the denoising network; finally, the generated secure image is determined based on the text embedding of the CLIP model and the random seed of the selected initial noise pixels, which can be represented as:

[0123] img = G(E(P));

[0124] Among them, img represents the generated image, and G represents the text graph model SD.

[0125] This application fine-tunes the parameters of the CLIP model to filter unsafe content in the text embedding stage, which specifically includes the following stages and steps:

[0126] (1) Dataset preparation stage

[0127] The negative sample dataset T (representing the dataset of negative samples) includes concepts of sensitive concepts or copyrighted objects that need to be removed (such as "nudity", "sex", "Van Gogh", etc.), which usually lead to unsafe image generation. Positive sample dataset (a dataset representing positive samples) is usually the antonym of negative samples, for example, "a naked woman" corresponds to "a woman wearing clothes". Specifically, "a naked woman" will generate an unqualified picture, while "wearing clothes" happens to be its antonym, avoiding the generation of unqualified images.

[0128] This application generates a positive sample dataset through a large language model. Given a negative sample t containing a violation description, the large language model is used to generate its corresponding positive sample. (benign sample). For concepts that need copyright protection or other concepts that need to be removed, the text embeddings of these words are mapped to the embeddings of placeholders. In this way, by removing these sensitive or protected concepts in the text encoding stage, it is possible to effectively avoid generating inappropriate or illegal image content.

[0129] In addition, a general dataset D containing common text prompts is used, such as sentences containing descriptions of daily life, such as "a person reading a book", to stabilize the text encoder and ensure that the text embeddings of other words do not deviate during the removal of sensitive concepts. This prevents the generated images from being semantically irrelevant to the input prompts, ensuring the generation ability and accuracy of the model.

[0130] (2) Fine-tuning the text encoder stage

[0131] like Figure 2 As shown, we start with two text encoders TE(·) (teacher model) and SE(·) (student model), both of which are initialized with the same pre-trained encoder weights used by the generative model. Fine-tuning with pre-trained weights works better than retraining a text encoder, and fine-tuning pre-trained weights works better than retraining a completely new text encoder. Retraining a new text encoder will almost certainly generate different embeddings that may not match the encoder currently used to guide image generation. Therefore, the entire generative model (text-image model) also needs to be retrained or at least adjusted to accommodate these new embeddings. Initializing and fine-tuning with pre-trained encoder weights ensures the consistency and stability of the embeddings, thereby improving the performance of the generative model and the quality of the generated images. This not only saves training time and computational resources, but also ensures that the generated images are highly semantically relevant to the input text prompt.

[0132] In the student-teacher model framework, in order to ensure that sensitive concepts in the generated text embeddings are mapped to corresponding security concepts, only the weights of the student model SE(·) are updated to map the text embeddings of sensitive concepts to corresponding security concepts, and the parameters of the teacher model TE(·) are kept unchanged. Specifically, the teacher model processes positive samples. Generate the corresponding positive text embedding The student model processes the negative sample t and generates the corresponding negative text embedding SE(t). The difference between the two embedding vectors is minimized through a suitable distance or similarity loss function, which can be expressed as

[0133] At the same time, in order to ensure that the quality and appearance of the generated images when the text-to-image model is used normally are similar to those when the original encoder is used and there is no significant deviation, a common dataset D containing a large amount of normal data is also used to ensure the practicality of the student encoder. This method is formulated as a bi-objective optimization problem, which aims to balance the effectiveness of removing sensitive concepts and maintaining the stability of normal input image generation. This optimization strategy not only improves the robustness and security of the model, but also ensures high standards in visual quality and semantic consistency of the generated images.

[0134] (3) Loss function design stage

[0135] In order to prevent the generation of unsafe concept images, the student encoder SE(·) (also called the student model SE(·)) needs to map unsafe concepts to the text embeddings of their corresponding safe concepts. To achieve this goal, this application uses cosine similarity to calculate the similarity of two text embeddings (positive sample text embedding and negative sample text embedding) and defines the following loss function:

[0136]

[0137] in, Denotes the negative sample embedding SE(t) generated by the student model and the positive sample embedding generated by the teacher model The cosine similarity of .

[0138] At the same time, in order to ensure that the student encoder does not affect the embedding effect of other texts, the following loss function is defined:

[0139]

[0140] where Cos(SE(d),TE(d)) represents the cosine similarity of the embedding vectors generated by the student model SE(·) and the teacher model TE(·) when processing sample d on the normal dataset D. By minimizing the loss function L1, we can ensure that the embedding generated by the student model is close to the embedding of the teacher model, thereby preserving the general utility of the encoder and ensuring the semantic consistency of the generated image with the normal input prompt.

[0141] In general, this application combines the above two loss functions to define the overall comprehensive loss function L:

[0142] L = L1 + β·L2;

[0143] Among them, β is a weight coefficient used to balance the loss function L1 that retains general utility and the loss function L2 that removes sensitive concepts.

[0144] (4) Student model text embedding stage

[0145] This application directly replaces the original text encoder with a trained student model. This replacement process does not require additional overhead and can be plug-and-play.

[0146] like Figure 3 As shown, the present application first generates a text embedding vector (which can be expressed as SE(P)) using the fine-tuned student model for the text prompt P, and then sends the text embedding vector to the cross attention layer of the denoising network to denoise the image, and uses the cross attention mechanism to guide the denoising process, and finally generates and outputs a safe image, which is expressed as:

[0147] img = G(SE(P));

[0148] Among them, SE() represents the fine-tuned student model and P represents the input text prompt.

[0149] It should be noted that, in order to distinguish the student model before and after fine-tuning, this application uses SE(·) to represent the student model before fine-tuning, and uses SE() to represent the student model after fine-tuning.

[0150] In summary, compared with existing methods such as concept erasing, the present application does not need to fine-tune the denoising parameters of the entire diffusion model, but only needs to fine-tune the text encoder. The advantage of this method is that the fine-tuning speed of the text encoder is much faster than the fine-tuning of the denoising network, and the required GPU resources are much less. Modifying the parameters of the denoising network may inadvertently have a significant impact on the quality of the generated image, while the present invention only fine-tunes the parameters of the text encoder, so it will not affect the quality of the generated image.

[0151] In addition, this application can flexibly respond to different types of sensitive content by dynamically adjusting text embedding. This method is not limited to a predefined list of sensitive words, but can also adapt to newly emerging sensitive words and adversarial samples by continuously updating training data.

[0152] On the other hand, compared with traditional text filters, when processing text input containing sensitive words, the present application avoids the problem of completely blocking text input by converting these sensitive words into embeddings of benign words. For example, for copyrighted content such as "Van Gogh-style starry sky", the text filter will directly block the output when it detects "Van Gogh", but the present invention will generate a picture of a star map without Van Gogh's style.

[0153] In summary, compared with existing methods such as concept erasing, the present application does not need to fine-tune the denoising parameters of the entire diffusion model, but only needs to fine-tune the text encoder. The advantage of this method is that the fine-tuning speed of the text encoder is much faster than the fine-tuning of the denoising network, and the required GPU resources are much less. Modifying the parameters of the denoising network may inadvertently have a significant impact on the quality of the generated image, while the present invention only fine-tunes the parameters of the text encoder, so it will not affect the quality of the generated image.

[0154] In addition, this application can flexibly respond to different types of sensitive content by dynamically adjusting text embedding. This method is not limited to a predefined list of sensitive words, but can also adapt to newly emerging sensitive words and adversarial samples by continuously updating training data.

[0155] On the other hand, compared with traditional text filters, when processing text input containing sensitive words, the present application avoids the problem of completely blocking text input by converting these sensitive words into embeddings of benign words. For example, for copyrighted content such as "Van Gogh-style starry sky", the text filter will directly block the output when it detects "Van Gogh", but the present invention will generate a picture of a star map without Van Gogh's style.

[0156] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0157] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technical personnel in this field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. A sensitive content filtering and defense method based on text embedding optimization of the text graph model, characterized by: The steps include: Prepare negative sample dataset T and normal dataset D; Based on the negative sample dataset T, a positive sample dataset is generated using a large language model Adjust the parameters of the text encoder and train it using the teacher-student model; Stabilize the text encoder using a common dataset D; Design a comprehensive loss function L, combine multiple data sets for training optimization and generate an adjusted student model; Use the adjusted student model for text embedding to generate and output the final security image; The teacher-student model includes a student model SE(·) and a teacher model TE(·).

2. The sensitive content filtering and defense method of the text-based graph model based on text embedding optimization according to claim 1 is characterized by: The negative sample dataset T includes sensitive concepts and concepts of copyrighted objects; the normal dataset D includes common text prompts; the positive sample dataset is the set of antonym sentences of negative samples in the negative sample dataset T.

3. The sensitive content filtering and defense method of the text-based graph model based on text embedding optimization according to claim 1 is characterized by: Generate positive sample dataset Specifically: Given a negative sample t containing a violation description; Use a large language model to generate positive samples corresponding to negative samples t For concepts of copyrighted objects or other concepts to be deleted, the text embeddings of the terms are mapped to the embeddings of the placeholders.

4. The sensitive content filtering and defense method of the text-based graph model based on text embedding optimization according to claim 1 is characterized by: Adjust the parameters of the text encoder, including the following: Both the student model SE(·) and the teacher model TE(·) are initialized using the same pre-trained encoder weights as the original text-generated graph model; Only the weights of the student model SE(·) are updated so that it maps the text embeddings of sensitive concepts to the corresponding security concepts; Keep the parameters of the teacher model TE(·) unchanged.

5. The sensitive content filtering and defense method of the text-based graph model based on text embedding optimization according to claim 1 is characterized by: The teacher-student model is used for training, specifically: Teacher model TE(·) processes positive samples And generate the corresponding positive sample text embedding The student model SE(·) processes the negative sample t and generates the corresponding negative sample text embedding SE(t).

6. The sensitive content filtering and defense method of the text-based graph model based on text embedding optimization according to claim 1 is characterized by: Design a comprehensive loss function L, including: Calculate positive sample text embedding using cosine similarity The similarity between the negative sample text embedding SE(t) and the second loss function L2 is defined as: in, represents the negative sample text embedding SE(t) generated by the student model SE(·) and the positive sample text embedding generated by the teacher model TE(·) The cosine similarity of .

7. The sensitive content filtering and defense method of the text-based graph model based on text embedding optimization according to claim 1 is characterized by: Also includes: Define the first loss function L1, expressed as: Among them, Cos(SE(d),TE(d)) represents the cosine similarity of the embedding vectors generated by the student model SE(·) and the teacher model TE(·) when processing the sampled common sample d on the common dataset D.

8. The sensitive content filtering and defense method of the text-based graph model based on text embedding optimization according to claim 1 is characterized by: Also includes: Combine the first loss function L1 and the second loss function L2 and define the comprehensive loss function L, expressed as: L = L1 + β·L2; Among them, β is a weight coefficient used to balance the first loss function L1 and the second loss function L2.

9. The sensitive content filtering and defense method of the text-based graph model based on text embedding optimization according to claim 1 is characterized by: The final safety image is generated using the adjusted student model, expressed as: img = G(SE(P)); Among them, SE() represents the adjusted student model and P represents the input text prompt.

Images