A backdoor defense method for text image diffusion model based on Ollivier-Ricci curvature
By constructing a conceptual database and utilizing Ollivier–Ricci curvature to detect feature space changes in the text-based image diffusion model, the problem of identifying backdoor attacks in the text-based image diffusion model was solved, achieving a backdoor defense effect with high accuracy and wide applicability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIHANG UNIV
- Filing Date
- 2026-03-06
- Publication Date
- 2026-06-05
AI Technical Summary
Existing technologies struggle to effectively identify and defend against backdoor attacks in text-based image diffusion models, especially due to insufficient generalization capabilities under multiple attack methods, leading to the spread of security vulnerabilities and risks.
By constructing a concept database, extracting feature vectors from the intermediate layers of the text encoder and denoising network, utilizing Ollivier–Ricci curvature to detect geometric changes in the feature space, constructing a K-nearest neighbor graph and calculating curvature, and performing anomaly detection in stages to identify backdoor inputs.
It improves the accuracy and generalization ability of backdoor detection, reduces the probability of false positives and false negatives, adapts to various backdoor attacks, and is suitable for practical deployment.
Smart Images
Figure CN122153891A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence security technology, and more specifically, to a backdoor defense method based on Ollivier–Ricci curvature-based text-to-image diffusion model. Background Technology
[0002] Diffusion models, as an important technique in generative artificial intelligence (GAI), have achieved significant application results in tasks such as image generation, image inpainting, and super-resolution reconstruction in recent years. Compared with traditional generative models (such as generative adversarial networks and variational autoencoders), diffusion models generate samples through a progressive denoising process, which has advantages such as stable training process, more controllable generation results, and higher quality. Therefore, they are widely used in generative tasks such as text-to-image processing.
[0003] However, with the widespread deployment of diffusion models in real-world scenarios, their security vulnerabilities have gradually become apparent, especially the issue of backdoor attacks. A backdoor attack refers to an attacker maliciously modifying the training process or data to implant a hidden triggering mechanism, i.e., a backdoor, into the model. When a diffusion model containing a backdoor receives input containing a specific trigger (i.e., a backdoor input), it generates the attacker's pre-defined target content; however, when receiving normal input without triggers, the model still exhibits the same generation behavior as a normal model, thus possessing strong stealth capabilities.
[0004] Backdoor attacks targeting text-based image diffusion models can be used to generate illegal, misleading, or harmful image content, posing serious social and security risks. Furthermore, since text-based image diffusion models are widely used in data synthesis and downstream model training, biases and anomalous behaviors introduced by malicious backdoors may propagate further to downstream applications through model output, causing even more severe harm. In addition, the training process of diffusion models typically relies on large-scale, high-quality data and expensive computing resources. In practical applications, model training is often completed by third parties or directly uses open-source models, which objectively increases the risk of backdoor attacks on diffusion models.
[0005] Due to the enormous input and output spaces of text-based image diffusion models, the number of recognizable text input combinations is extremely large. Backdoor defense methods that attempt to directly locate and eliminate specific "trigger-target image" associations at the model level are practically unrealistic. Existing defense techniques focus on input-level backdoor defense, i.e., detecting backdoors by determining whether the input text prompt contains triggers. However, such methods typically rely on specific anomalies triggered by backdoor inputs for detection, such as abnormal attention distribution or abnormal attention change rates. This makes the defense highly sensitive to the attack method, difficult to adapt to backdoor attacks employing different triggering mechanisms, and has limited overall generalization ability.
[0006] Therefore, how to design a backdoor input detection method for text image diffusion models with higher detection accuracy and stronger generalization ability, so that it can be applied to a variety of backdoor attack scenarios, has become a key technical problem in current research and a technical challenge that needs to be solved by those skilled in the art. Summary of the Invention
[0007] In view of this, this invention provides a backdoor defense method for text-based graph diffusion models based on Ollivier–Ricci curvature. This method effectively identifies backdoor inputs containing triggers by detecting geometric changes in the feature space of the text encoder and the feature space of the intermediate layer of the denoising network in the diffusion model, thereby improving the security of the diffusion model and avoiding potential backdoor attack risks. Compared with existing methods, the proposed method makes better use of the generation mechanism of the text-based graph diffusion model, and the proposed detection index is weakly correlated with the triggering mechanism of backdoor attacks, thus improving the detection accuracy and overall generalization ability of the method. This invention also has good scalability, is applicable to various diffusion models and samplers, and has a short running time, making it suitable for practical deployment.
[0008] To achieve the above objectives, the present invention adopts the following technical solution:
[0009] Step S1: Establish a concept database, which stores multiple predefined concepts and their corresponding text encoder feature vectors and denoising network intermediate layer feature vectors. The feature vectors are obtained by encoding the text prompts associated with each predefined concept and forward propagating the denoising network.
[0010] Step S2: Input the text prompt to be detected into the diffusion model pipeline to obtain its corresponding text encoder feature vector and denoising network intermediate layer feature vector; perform semantic similarity matching between the text prompt to be detected and the text prompts stored in the concept database to obtain the reference feature vector associated with the matched text prompt; and perform regularization processing based on the reference feature vector and the feature vector to be detected.
[0011] Step S3: Construct features based on the text encoder feature vector of the text prompt to be detected and its corresponding reference text feature vector. Nearest neighbor graph, wherein each node of the graph corresponds to the text encoder feature vector of the text prompt to be detected or the reference text prompt; calculate the Ollivier-Ricci curvature of each undirected edge in the nearest neighbor graph; node curvature of each node is obtained by statistical aggregation of the edge curvature; and based on the node curvature, it is determined whether the node curvature corresponding to the text prompt to be detected exceeds a preset abnormal threshold.
[0012] Step S4: If the text encoder features of the text prompt to be detected are not determined to be abnormal, then the intermediate layer feature vectors of the denoising network for the text prompt to be detected and the matching reference item are processed in... Pooling is performed at each time step to construct the corresponding global feature vector; for the global feature at each time step, a curvature representation is constructed based on the same curvature calculation method as in step S3, and the proportion of the text prompt to be detected exceeding the preset abnormal threshold in the global curvature representation at each time step is counted; if the proportion exceeds the preset judgment proportion, the global feature is determined to be abnormal.
[0013] Step S5: If the text encoder features or global curvature representation of the text prompt to be detected are determined to be abnormal, then in At each time step, the feature vectors of the intermediate layer of the denoising network for the text prompt to be detected and the matching reference are spatially divided into blocks to obtain local feature vectors. For each local feature vector at each time step, a corresponding local curvature representation is constructed based on the same curvature calculation method as in step S3, and the proportion of the node curvature corresponding to the text prompt to be detected in all local curvature representations that exceeds a preset abnormal threshold is counted. If the proportion exceeds a preset backdoor determination threshold, it is determined that the text prompt to be detected contains a backdoor trigger to achieve backdoor defense.
[0014] Step S1 involves establishing a concept database. The generation of this database specifically includes:
[0015] Step S1.1: Determine the diffusion model architecture for the text image, including its text encoder, denoising network, and the text encoder output layer and denoising network intermediate layers for feature extraction;
[0016] Step S1.2: Construct a set of predefined concepts, and for each predefined concept, use a pre-trained large language model to generate multiple text prompts describing the concept, forming a text prompt library associated with the predefined concept;
[0017] Step S1.3: Input the text prompts into the diffusion model respectively, perform forward generation to obtain the corresponding generated images; use a multimodal large model to perform semantic-visual consistency evaluation on the generated images, and filter out text prompts with qualified image quality; input the qualified text prompts into the diffusion model again, and extract their corresponding text encoder feature vectors and denoising network intermediate layer feature vectors during the forward propagation process.
[0018] Step S1.4: Associate and store the extracted text encoder feature vector, the intermediate layer feature vector of the denoising network, and the corresponding predefined concepts and original text prompts to construct the concept database.
[0019] Step S2 involves extracting the feature vector of the input to be detected, loading the reference feature vector, and performing regularization processing. This step specifically includes:
[0020] Step S2.1: Input the text prompt to be detected into the diffusion model pipeline, perform forward propagation to extract its corresponding text encoder feature vector and denoising network intermediate layer feature vector as the feature to be detected;
[0021] Step S2.2: Based on the semantic similarity between the text prompt to be detected and each text prompt stored in the concept database, select several matching text prompts with the highest similarity, and use the text encoder feature vector and the intermediate layer feature vector of the denoising network stored in the concept database as a reference feature vector set; calculate the mean, variance and correlation matrix of the reference feature vector set as statistics;
[0022] Step S2.3: Based on the statistics, perform regularization processing on the text encoder feature vector, the intermediate layer feature vector of the denoising network, and the corresponding reference feature vector in the features to be detected.
[0023] In step S3, the text encoder feature vector is mapped and Ollivier-Ricci curvature is calculated. Anomalies are detected based on the curvature calculation results. This anomaly detection method specifically includes:
[0024] Step S3.1: In the feature space formed by the text encoder feature vector of the text prompt to be detected and its corresponding reference text feature vector, construct a feature space based on Euclidean distance or cosine distance. The nearest neighbor graph, wherein each node of the graph corresponds to the text encoder feature vector of the text prompt to be detected or the reference text prompt;
[0025] Step S3.2: Calculate the The Ollivier-Ricci curvature of each undirected edge in the nearest neighbor graph is calculated, and the curvature of the edge associated with each node is statistically aggregated to obtain the node curvature; wherein the statistical aggregation adopts at least one of the following methods: mean, quantile, maximum or weighted statistics.
[0026] Step S3.3: Compare the node curvature corresponding to the text prompt to be detected with the distribution of node curvature corresponding to the reference text prompt. If the node curvature of the text prompt to be detected exceeds the preset quantile threshold of the distribution, it is determined that its text features are abnormal.
[0027] In step S4, anomaly detection based on Ollivier-Ricci curvature is performed on the global feature vector of the intermediate layer of the denoising network. This anomaly detection method specifically includes:
[0028] Step S4.1: In At each preset time step, the feature vectors of the intermediate layer of the denoising network for the text prompt to be detected and the matching reference item are obtained respectively, forming a shape of... The intermediate feature vector; for each time step, the feature vector in the spatial dimension ( Pooling is performed to obtain The shape is The global feature vector, where This indicates the total number of the text prompt to be detected and the matching reference items. The number of feature channels, and These represent the height and width of the feature map, respectively; the pooling method is either average pooling or max pooling.
[0029] Step S4.2: For the global feature vector obtained in step S4.1 at each time step, construct a new feature vector using the same curvature calculation method as in step S3. The nearest neighbor graph is used to calculate the curvature representation of each node, and finally, the result is obtained. A global curvature representation is established; the distribution of global node curvature corresponding to the text prompt to be detected is compared with that of global node curvature corresponding to the reference text prompt, and the proportion of the text prompt to be detected exceeding the preset abnormal threshold in the global curvature representation at each time step is counted; if the proportion exceeds the preset judgment proportion, the global feature is determined to be abnormal.
[0030] In step S5, anomaly detection based on Ollivier-Ricci curvature is performed on the local features of the intermediate layers of the denoising network. This anomaly detection method specifically includes:
[0031] Step S5.1: If the text encoder features or global curvature representation of the text prompt to be detected are determined to be abnormal, then in At each preset time step, the feature vectors of the intermediate layer of the denoising network for the text prompt to be detected and the matching reference item are obtained respectively, forming a shape of... The intermediate feature vector; for each time step, the feature vector is divided into two segments in both the height and width directions, resulting in four local regions, yielding four sets of shapes. The local feature vectors are then pooled in the spatial dimension to obtain four spatial regions. Local features at each time step, totaling The shape is The local feature vectors, where This indicates the total number of the text prompt to be detected and the matching reference items. The number of feature channels, and These represent the height and width of the feature map, respectively; the pooling method is either average pooling or max pooling.
[0032] Step S5.2: For each set of local features, construct the corresponding local features using the same curvature calculation method as in Step S3. The nearest neighbor graph is used to calculate the local curvature representation of each node, and finally, the result is obtained. A local curvature characterization;
[0033] Step S5.3: Compare the distribution of local node curvature corresponding to the text prompt to be detected with that of the local node curvature corresponding to the reference text prompt, and count the proportion of the node curvature corresponding to the text prompt to be detected that exceeds a preset abnormal threshold among all local curvature representations; if the proportion exceeds a preset backdoor determination threshold, then determine that the text prompt to be detected contains a backdoor trigger, so as to achieve backdoor defense.
[0034] The beneficial effects of this invention are as follows: In terms of detection accuracy, by constructing a K-nearest neighbor graph in the feature space and introducing Ollivier-Ricci curvature, this invention characterizes the geometric changes in the feature space of the text encoder and the intermediate layer feature space of the denoising network in the diffusion model. This effectively captures local structural anomalies caused by backdoor triggers, thereby improving the detection accuracy of backdoor inputs and reducing the probability of false positives and false negatives. In terms of detection efficiency, this invention adopts a phased detection strategy. It first performs rapid screening of global features at a small number of time steps, and only performs fine-grained local detection at more time steps when anomalies are detected. This effectively reduces overall computational overhead and improves detection efficiency while ensuring detection performance, making it suitable for practical deployment scenarios. In terms of overall generalization ability, this invention does not rely on specific trigger forms or specific attack methods, but rather makes judgments based on changes in the geometric structure of the feature space. It can adapt to various types of backdoor attacks and has good generalization ability against unknown or variant attacks, enhancing the security of the text-based graph diffusion model in complex application scenarios. Attached Figure Description
[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0036] Figure 1 This is a flowchart illustrating a backdoor defense method based on the Ollivier–Ricci curvature-based Wensheng graph diffusion model provided by the present invention.
[0037] Figure 2 This is a schematic diagram of the overall framework of a backdoor defense method based on the Ollivier–Ricci curvature Wensheng graph diffusion model provided by the present invention.
[0038] Figure 3 This is a schematic diagram illustrating the detection accuracy of three defense algorithms (DAA, FFT, and the method proposed in this patent) under five different attack scenarios in an embodiment of the present invention. Detailed Implementation
[0039] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0040] Example 1
[0041] like Figure 1 and Figure 2 This invention discloses a backdoor defense method for a text image diffusion model based on Ollivier–Ricci curvature, specifically including the following steps:
[0042] Step S1: Establish a concept database, which stores multiple predefined concepts and their corresponding text encoder feature vectors and denoising network intermediate layer feature vectors. The feature vectors are obtained by encoding the text prompts associated with each predefined concept and forward propagating them through the denoising network.
[0043] Step S1.1: Determine the diffusion model pipeline for the text image, including its text encoder, denoising network, and the text encoder output layer and denoising network intermediate layers for feature extraction. In this embodiment, the text image diffusion model pipeline uses Stable Diffusion 1.5, which includes a text encoder using a pre-trained CLIP model and a denoising network based on the Unet architecture. Text features are taken from the output of the text encoder. The text embeddings output by the text encoder are averaged within the effective mask range to obtain the text encoder feature vector corresponding to the text prompt. The intermediate features of the diffusion model use the output of the midblock layer of the denoising network as feature vectors reflecting the semantic information of the generation process. During sampling, the number of sampling steps is num_steps=24; the quality control parameters are guidance_scale=7.0 and guidance_rescale=0.15.
[0044] Step S1.2: Construct a predefined concept set, and for each predefined concept, use a pre-trained large language model to generate multiple text prompts describing that concept, forming a text prompt library associated with the predefined concept. In this embodiment, the predefined concept set is constructed around the common semantic concepts "dog" and "cat". For each concept, DeepSeek-R1 is used to generate 600 semantically diverse and naturally expressed text prompts.
[0045] Step S1.3: Input the text prompts into the diffusion model pipeline and perform forward generation to obtain corresponding generated images; use a multimodal large model to evaluate the semantic-visual consistency of the generated images and filter out text prompts with acceptable image quality; input the acceptable text prompts back into the diffusion model, and extract their corresponding text encoder feature vectors and denoising network intermediate layer feature vectors during forward propagation. In this embodiment, the generated images are introduced into the multimodal large model Qwen-VL 2.5 for quality screening. The screening criteria include whether the generated images are semantically consistent with the input text and the stability of the generation results under four different random seeds. For text prompts that pass the quality screening, input them back into the diffusion model for sampling, and extract the corresponding text encoder feature vectors and denoising network intermediate layer feature vectors during the sampling process.
[0046] Step S1.4: Associate and store the extracted text encoder feature vector, the intermediate layer feature vector of the denoising network, and the corresponding predefined concepts and original text prompts to construct the concept database.
[0047] Step S2: Input the text prompt to be detected into the diffusion model pipeline to obtain its corresponding text encoder feature vector and denoising network intermediate layer feature vector; match the semantic similarity between the text prompt to be detected and the text prompts stored in the concept database to obtain the reference feature vector associated with the matched text prompt; and perform regularization processing together with the reference feature vector and the feature vector to be detected.
[0048] Step S2.1: Input the text prompt to be detected into the diffusion model pipeline, perform forward propagation to extract its corresponding text encoder feature vector and denoising network intermediate layer feature vector as the features to be detected. In this embodiment, the text prompt to be detected is a text prompt related to "dog" and "cat", corresponding to the concept database.
[0049] Step S2.2: Based on the semantic similarity between the text prompt to be detected and the text prompts stored in the concept database, select several matching text prompts with the highest similarity, and use the text encoder feature vector and the intermediate layer feature vector of the denoising network stored in the concept database as a reference feature vector set; calculate the mean, variance, and correlation matrix of the reference feature vector set as statistics. In this embodiment, the text prompt is fed into the language model Sentence-BERT to obtain the text embedding vector, and then text prompt matching is performed based on cosine similarity distance.
[0050] Step S2.3: Based on the statistics, perform regularization processing on the text encoder feature vector, the denoising network intermediate layer feature vector, and the corresponding reference feature vector in the features to be detected. In this embodiment, L2 normalization is used for regularization processing.
[0051] Step S3: Construct a feature K-nearest neighbor graph based on the text encoder feature vector of the text prompt to be detected and its corresponding reference text feature vector, wherein each node of the graph corresponds to the text encoder feature vector of the text prompt to be detected or the reference text prompt; calculate the... Ollivier-Ricci curvature of each undirected edge in the nearest neighbor graph; node curvature of each node is obtained by statistical aggregation of the edge curvature; and based on the node curvature, it is determined whether the node curvature corresponding to the text prompt to be detected exceeds a preset abnormal threshold.
[0052] Step S3.1: In the feature space formed by the text encoder feature vector of the text prompt to be detected and its corresponding reference text feature vector, construct a feature space based on Euclidean distance or cosine distance. A nearest neighbor graph, wherein each node of the graph corresponds to a text encoder feature vector of the text prompt to be detected or the reference text prompt. In this embodiment... .
[0053] Step S3.2: Calculate the Ollivier-Ricci curvature of each undirected edge in the K-nearest neighbor graph, and aggregate the statistics of the edge curvature associated with each node to obtain the node curvature. In this embodiment, the statistics used are the 20th percentiles of the edge curvature distribution.
[0054] Step S3.3: Compare the node curvature corresponding to the text prompt to be detected with the distribution of node curvature corresponding to the reference text prompt. If the node curvature of the text prompt to be detected exceeds the preset quantile threshold of the distribution, it is determined that its text features are abnormal. In this embodiment, the preset quantile is the 95th quantile of the node curvature distribution of the reference text.
[0055] Step S4: If the text encoder features of the text prompt to be detected are not determined to be abnormal, then the intermediate layer feature vectors of the denoising network for the text prompt to be detected and the matching reference item are processed in... Pooling is performed at each time step to construct the corresponding global feature vector; for the global feature at each time step, a curvature representation is constructed based on the same curvature calculation method as in step S3, and the proportion of the text prompt to be detected exceeding the preset abnormal threshold in the global curvature representation at each time step is counted; if the proportion exceeds the preset judgment proportion, the global feature is determined to be abnormal.
[0056] Step S4.1: In At each preset time step, the feature vectors of the intermediate layer of the denoising network for the text prompt to be detected and the matching reference item are obtained respectively, forming a shape of... The intermediate feature vector; for each time step, the feature vector in the spatial dimension ( Pooling is performed to obtain The shape is The global feature vector, where This indicates the total number of the text prompt to be detected and the matching reference items. The number of feature channels, and These represent the height and width of the feature map, respectively; the pooling method is either average pooling or max pooling. In this embodiment, average pooling is used. The preset time step is . .
[0057] Step S4.2: For the global feature vector obtained in step S4.1 at each time step, construct a new feature vector using the same curvature calculation method as in step S3. The nearest neighbor graph is used to calculate the curvature representation of each node, and finally, the result is obtained. A global curvature representation is performed; the distribution of global node curvature corresponding to the text prompt to be detected is compared with that of global node curvature corresponding to the reference text prompt, and the proportion of the global curvature representation of the text prompt to be detected exceeding a preset anomaly threshold at each time step is calculated; if the proportion exceeds a preset judgment proportion, the global feature is determined to be abnormal. In this embodiment, the preset anomaly threshold is the 95th percentile of the global node curvature distribution corresponding to the reference text prompt, and the preset judgment proportion is 25%.
[0058] Step S5: If the text encoder features or global curvature representation of the text prompt to be detected are determined to be abnormal, then in At each time step, the feature vectors of the intermediate layer of the denoising network for the text prompt to be detected and the matching reference are spatially divided into blocks to obtain local feature vectors. For each local feature vector at each time step, a corresponding local curvature representation is constructed based on the same curvature calculation method as in step S3, and the proportion of the node curvature corresponding to the text prompt to be detected in all local curvature representations that exceeds a preset abnormal threshold is counted. If the proportion exceeds a preset backdoor determination threshold, it is determined that the text prompt to be detected contains a backdoor trigger to achieve backdoor defense.
[0059] Step S5.1: If the text encoder features or global curvature representation of the text prompt to be detected are determined to be abnormal, then in At each preset time step, the feature vectors of the intermediate layer of the denoising network for the text prompt to be detected and the matching reference item are obtained respectively, forming a shape of... The intermediate feature vector; for each time step, the feature vector is divided into two segments in both the height and width directions, resulting in four local regions, yielding four sets of shapes. The local feature vectors are then pooled in the spatial dimension to obtain four spatial regions. Local features at each time step, totaling The shape is The local feature vectors, where This indicates the total number of the text prompt to be detected and the matching reference items. The number of feature channels, and These represent the height and width of the feature map, respectively; the pooling method is either average pooling or max pooling. In this embodiment, The preset time step is t= . .
[0060] Step S5.2: For each set of local features, construct the corresponding local features using the same curvature calculation method as in Step S3. The nearest neighbor graph is used to calculate the local curvature representation of each node, and finally, the result is obtained. A local curvature characterization.
[0061] Step S5.3: Compare the distribution of local node curvature corresponding to the text prompt to be detected with that of the local node curvature corresponding to the reference text prompt. Calculate the proportion of nodes whose curvature exceeds a preset anomaly threshold among all local curvature representations. If the proportion exceeds a preset backdoor detection threshold, the text prompt to be detected is determined to contain a backdoor trigger, thus achieving backdoor defense. In this embodiment, the preset anomaly threshold is the 95th percentile of the local node curvature distribution corresponding to the reference text prompt. In the backdoor detection process, the threshold design used in this embodiment is as follows: if two or more local blocks exhibit curvature anomalies in a time step, that time step is considered abnormal; if more than 1 / 4 of the time steps are abnormal, the input to be detected is considered to contain a trigger.
[0062] Figure 3This paper demonstrates a performance comparison between the proposed method and two state-of-the-art backdoor defense methods for raw image diffusion models (DAA and FFT). This embodiment tests five state-of-the-art raw image diffusion model backdoor attack methods (Rickrolling, TwT, BadT2I, VillanDiff, and EvilEdit). In all five attack scenarios, the proposed method achieves the highest backdoor input detection success rate, proving the effectiveness of the proposed method.
[0063] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A backdoor defense method based on Ollivier–Ricci curvature-based text image diffusion model, characterized in that, The method is applicable to the diffusion model of text-to-image generation and is used to detect whether the text prompts obtained by the diffusion model contain backdoor triggers, so as to achieve backdoor defense; the method includes the following steps: Step S1: Establish a concept database, which stores multiple predefined concepts and their corresponding text encoder feature vectors and denoising network intermediate layer feature vectors. The feature vectors are obtained by encoding the text prompts associated with each predefined concept and forward propagating the denoising network. Step S2: Input the text prompt to be detected into the diffusion model pipeline to obtain its corresponding text encoder feature vector and denoising network intermediate layer feature vector; match the semantic similarity between the text prompt to be detected and the text prompts stored in the concept database to obtain the reference feature vector associated with the matched text prompt; and perform regularization processing based on the reference feature vector and the feature vector to be detected. Step S3: Construct features based on the text encoder feature vector of the text prompt to be detected and its corresponding reference text feature vector. Nearest neighbor graph, wherein each node of the graph corresponds to the text encoder feature vector of the text prompt to be detected or the reference text prompt; calculate the Ollivier-Ricci curvature of each undirected edge in the nearest neighbor graph; node curvature of each node is obtained by statistical aggregation of the edge curvature; and based on the node curvature, it is determined whether the node curvature corresponding to the text prompt to be detected exceeds a preset abnormal threshold. Step S4: If the text encoder features of the text prompt to be detected are not determined to be abnormal, then the intermediate layer feature vectors of the denoising network for the text prompt to be detected and the matching reference item are processed in... Pooling is performed at each time step to construct the corresponding global feature vector; for the global feature at each time step, a curvature representation is constructed based on the same curvature calculation method as in step S3, and the proportion of the text prompt to be detected exceeding the preset abnormal threshold in the global curvature representation at each time step is counted; if the proportion exceeds the preset judgment proportion, the global feature is determined to be abnormal. Step S5: If the text encoder features or global curvature representation of the text prompt to be detected are determined to be abnormal, then in At each time step, the feature vectors of the intermediate layer of the denoising network for the text prompt to be detected and the matching reference are spatially divided into blocks to obtain local feature vectors. For each local feature vector at each time step, a corresponding local curvature representation is constructed based on the same curvature calculation method as in step S3, and the proportion of the node curvature corresponding to the text prompt to be detected in all local curvature representations that exceeds a preset abnormal threshold is counted. If the proportion exceeds a preset backdoor determination threshold, it is determined that the text prompt to be detected contains a backdoor trigger to achieve backdoor defense.
2. The backdoor defense method for the text-based image diffusion model based on Ollivier–Ricci curvature according to claim 1, characterized in that, Step S1 includes the following sub-steps: Step S1.1: Determine the diffusion model architecture for the text image, including its text encoder, denoising network, and the text encoder output layer and denoising network intermediate layers for feature extraction; Step S1.2: Construct a set of predefined concepts, and for each predefined concept, use a pre-trained large language model to generate multiple text prompts describing the concept, forming a text prompt library associated with the predefined concept; Step S1.3: Input the text prompts into the diffusion model respectively, perform forward generation to obtain the corresponding generated images; use a multimodal large model to perform semantic-visual consistency evaluation on the generated images, and filter out text prompts with qualified image quality; input the qualified text prompts into the diffusion model again, and extract their corresponding text encoder feature vectors and denoising network intermediate layer feature vectors during the forward propagation process. Step S1.4: Associate and store the extracted text encoder feature vector, the denoising network intermediate layer feature vector, and the corresponding predefined concepts and original text prompts to construct the concept database.
3. The backdoor defense method based on Ollivier–Ricci curvature for textual image diffusion models according to claim 1, characterized in that, Step S2 includes the following sub-steps: Step S2.1: Input the text prompt to be detected into the diffusion model pipeline, perform forward propagation to extract its corresponding text encoder feature vector and denoising network intermediate layer feature vector as the feature to be detected; Step S2.2: Based on the semantic similarity between the text prompt to be detected and each text prompt stored in the concept database, select several matching text prompts with the highest similarity, and use the text encoder feature vector and the intermediate layer feature vector of the denoising network stored in the concept database as a reference feature vector set; calculate the mean, variance and correlation matrix of the reference feature vector set as statistics; Step S2.3: Based on the statistics, perform regularization processing on the text encoder feature vector, the intermediate layer feature vector of the denoising network, and the corresponding reference feature vector in the features to be detected.
4. The backdoor defense method based on Ollivier–Ricci curvature for textual image diffusion models according to claim 1, characterized in that, Step S3 includes the following sub-steps: Step S3.1: In the feature space formed by the text encoder feature vector of the text prompt to be detected and its corresponding reference text feature vector, construct a feature space based on Euclidean distance or cosine distance. The nearest neighbor graph, wherein each node of the graph corresponds to the text encoder feature vector of the text prompt to be detected or the reference text prompt; Step S3.2: Calculate the The Ollivier-Ricci curvature of each undirected edge in the nearest neighbor graph is calculated, and the curvature of the edge associated with each node is statistically aggregated to obtain the node curvature; wherein the statistical aggregation adopts at least one of the following methods: mean, quantile, maximum or weighted statistics. Step S3.3: Compare the node curvature corresponding to the text prompt to be detected with the distribution of node curvature corresponding to the reference text prompt. If the node curvature of the text prompt to be detected exceeds the preset quantile threshold of the distribution, it is determined that its text features are abnormal.
5. The backdoor defense method based on Ollivier–Ricci curvature for textual image diffusion models according to claim 1, characterized in that, Step S4 includes the following sub-steps: Step S4.1: In At each preset time step, the feature vectors of the intermediate layer of the denoising network for the text prompt to be detected and the matching reference item are obtained respectively, forming a shape of... The intermediate feature vector; for each time step, the feature vector in the spatial dimension ( Pooling is performed to obtain The shape is The global feature vector, where This indicates the total number of the text prompt to be detected and the matching reference items. The number of feature channels, and These represent the height and width of the feature map, respectively; the pooling method is either average pooling or max pooling. Step S4.2: For the global feature vector obtained in step S4.1 at each time step, construct a new feature vector using the same curvature calculation method as in step S3. The nearest neighbor graph is used to calculate the curvature representation of each node, and finally, the result is obtained. A global curvature representation is established; the distribution of global node curvature corresponding to the text prompt to be detected is compared with that of global node curvature corresponding to the reference text prompt, and the proportion of the text prompt to be detected exceeding the preset abnormal threshold in the global curvature representation at each time step is counted; if the proportion exceeds the preset judgment proportion, the global feature is determined to be abnormal.
6. The backdoor defense method for the text-based image diffusion model based on Ollivier–Ricci curvature according to claim 1, characterized in that, Step S5 includes the following sub-steps: Step S5.1: If the text encoder features or global curvature representation of the text prompt to be detected are determined to be abnormal, then in At each preset time step, the feature vectors of the intermediate layer of the denoising network for the text prompt to be detected and the matching reference item are obtained respectively, forming a shape of... The intermediate feature vector; for each time step, the feature vector is divided into two segments in both the height and width directions, resulting in four local regions, yielding four sets of shapes. The local feature vectors are then pooled in the spatial dimension to obtain four spatial regions. Local features at each time step, totaling The shape is The local feature vectors, where This indicates the total number of the text prompt to be detected and the matching reference items. The number of feature channels, and These represent the height and width of the feature map, respectively; the pooling method is either average pooling or max pooling. Step S5.2: For each set of local features, construct the corresponding local features using the same curvature calculation method as in Step S3. The nearest neighbor graph is used to calculate the local curvature representation of each node, and finally, the result is obtained. A local curvature characterization; Step S5.3: Compare the distribution of local node curvature corresponding to the text prompt to be detected with that of the local node curvature corresponding to the reference text prompt, and count the proportion of the node curvature corresponding to the text prompt to be detected that exceeds a preset abnormal threshold among all local curvature representations; if the proportion exceeds a preset backdoor determination threshold, then determine that the text prompt to be detected contains a backdoor trigger, so as to achieve backdoor defense.