Backdoor attack method for diffusion bridge model

Abstract

The invention discloses a backdoor attack method for a diffusion bridge model, and the method comprises the following steps: constructing a data set which comprises backdoor pairing data, and the backdoor pairing refers to the pairing of an image with a trigger and a predefined backdoor image; the data set is adopted to train a hybrid expert model, the hybrid expert model comprises a plurality of expert models, and each expert model is trained on different generation tasks; and generating an image by adopting the trained model. According to the diffusion bridge model for the graph-to-graph task, different generation tasks can be coped with, normal graph-to-graph tasks and back door attack tasks can be taken into consideration, stability and reliability detection of the diffusion model can be achieved, and a new index is provided for model performance detection.

Description

Technical Field

[0001] The invention relates to a backdoor attack method for a diffusion bridge model, and belongs to the technical field of image generation. Background Art

[0002] Image generation technology, especially the use of diffusion models, can generate semantically consistent high-quality images based on existing images or text prompts.

[0003] Backdoor attacks are sometimes necessary for diffusion models. Backdoor attack methods not only provide methods and environments for diffusion model security research, but can also be used to reveal weak links in large models during training, fine-tuning, and inference, such as third-party data dependence and easy manipulation of intermediate layer parameters. This provides improvement directions for model design and deployment and helps explore new training strategies.

[0004] In a backdoor attack scenario, an attacker can secretly implant a backdoor into the diffusion model, and then manipulate the diffusion model to generate the desired image by adding triggers to the input.

[0005] In addition, backdoor attacks can be applied in specific environments to detect the stability and reliability of the diffusion model.

[0006] Existing diffusion model backdoor attacks mainly target unconditional diffusion models and text-generated image diffusion models. However, there is a lack of attack methods for the diffusion bridge model of image-generated image tasks, that is, the diffusion model that accepts images as input to generate images, and it is impossible to dynamically adjust the attack strategy according to the application scenario.

[0007] Therefore, it is necessary to conduct a more in-depth study of existing backdoor attack methods to solve the above problems. Summary of the Invention

[0008] To overcome the above problems, we conducted in-depth research and proposed a backdoor attack method for the diffusion bridge model, which includes the following steps:

[0009] S1. Constructing a data set, wherein the data set includes backdoor pairing data, wherein the backdoor pairing refers to pairing an image with a trigger with a predefined backdoor image;

[0010] S2. Using the dataset to train a hybrid expert model, the hybrid expert model includes multiple expert models, each expert model is trained on a different generation task;

[0011] S3. Generate images using the trained model.

[0012] In a preferred embodiment, in S1, the data set further includes normal image-generated image pairing data, where the normal image-generated image refers to a pairing of an image without a trigger and a generated image;

[0013] In the backdoor pairing data, the trigger is a mark loaded on the normal image.

[0014] In a preferred embodiment, the backdoor pairing is represented as:

[0015]

[0016] Among them, δ represents the trigger, which is mixed into the normal input image Generated a backdoor image with a trigger M is a 0-1 matrix.

[0017] In a preferred embodiment, in S2, the training of the hybrid expert model includes two stages: the first stage is to train different expert models separately, and the second stage is to combine the trained expert models;

[0018] The expert model is a diffusion model.

[0019] In a preferred embodiment, the second stage uses the routing model for combination, and the result of the combination is used as the output of the hybrid expert model.

[0020] In a preferred embodiment, the output of the hybrid expert model is expressed as:

[0021]

[0022] Among them, ∈ Mix represents the mixed expert model, t represents the time step, represents the input of the expert model at the tth time step, θ Mix represents the model parameters of the hybrid expert model, N represents the total number of expert models trained on normal and raw image pairing data, represents the jth expert model among the expert models trained on normal image and raw image pairing data, Represents an expert model The weight of Represents an expert model Model parameters; M represents the total number of expert models trained with backdoor paired data, represents the i-th expert model in the expert model trained with backdoor paired data, Represents an expert model The weight of Represents an expert model The model parameters.

[0023] In a preferred embodiment, in the combination, the weights are constrained to balance the contributions of different expert models to the hybrid expert model.

[0024] In a preferred embodiment, there is further step S4, using the model obtained in step S3 as a post-attack model;

[0025] The expert model is trained using a dataset with only normal image and raw image pairing data to obtain the original diffusion model. The performance difference between the original diffusion model and the attacked model is obtained, and the stability of the expert model is evaluated based on the size of the difference.

[0026] The present invention also provides an electronic device, comprising:

[0027] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any one of the above methods.

[0028] The present invention also provides a computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute any one of the above methods.

[0029] The beneficial effects of the present invention include:

[0030] (1) The diffusion bridge model for graph-to-graph tasks can cope with different generation tasks and can take into account both normal graph-to-graph tasks and backdoor attack tasks;

[0031] (2) During training, the two-stage training strategy alleviates the conflict between different generation tasks and improves the quality of image generation;

[0032] (3) It can realize the stability and reliability detection of the diffusion model and provide a new indicator for model performance detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 The figure is a flow chart of a backdoor attack method of a diffusion bridge model according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be described in further detail below with reference to the accompanying drawings and examples, through which the features and advantages of the present invention will become more clearly understood.

[0035] The word "exemplary" is used exclusively herein to mean "serving as an example, example, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0036] According to the present invention, a backdoor attack method for a diffusion bridge model is provided, such as Figure 1 As shown, the following steps are included:

[0037] S1. Constructing a data set, wherein the data set includes backdoor pairing data, wherein the backdoor pairing refers to pairing an image with a trigger with a predefined backdoor image;

[0038] S2. Using the dataset to train a hybrid expert model, the hybrid expert model includes multiple expert models, each expert model is trained on a different generation task;

[0039] S3. Generate images using the trained model.

[0040] In S1, the data set also includes normal image and raw image pairing data, and the normal image and raw image refer to the pairing of the image without trigger and the generated image.

[0041] Preferably, in S1, the normal image and raw image pairing data have a method of obtaining multiple task data.

[0042] In the present invention, the specific type of the task data is not limited, and those skilled in the art can freely set it according to actual needs. For example, the task data can be super-resolution pairing task data: pairing a low-resolution image with a high-resolution image of the same content to form a training sample;

[0043] For another example, the task data may be image restoration pairing task data: pairing a damaged image with a complete image of the same content to form a training sample.

[0044] In a preferred embodiment, in the backdoor pairing data, the trigger is set as a mark loaded on a normal image, generally a subtle mark in an inconspicuous position on the image, such as adding a small black square in the lower right corner of the image.

[0045] The predefined backdoor image can be set by those skilled in the art according to actual needs, for example, to a malicious and illegal image.

[0046] The backdoor pairing is represented as:

[0047]

[0048] Among them, δ represents the trigger, which is mixed into the normal input image Generated a backdoor image with a trigger M is a 0-1 matrix that represents the shape and size of the trigger and its position covered on the normal image.

[0049] In S2, the training of the hybrid expert model includes two stages. In the first stage, different expert models are trained separately to maximize the performance of the expert models on different tasks.

[0050] In the second stage, the trained expert models are combined and the contribution of each expert model to the final output is balanced through a training strategy of weight redistribution.

[0051] In the present invention, the expert model may be any diffusion model, for example, a diffusion Schrödinger bridge model.

[0052] According to the present invention, an expert model is trained for each task in the data set. For example, when the normal image and raw image pairing data contain super-resolution pairing task data and image restoration pairing task data, and the backdoor pairing data contain face tampering task data and illegal image data, an expert model is trained for the super-resolution pairing task data, an expert model is trained for the image restoration pairing task data, an expert model is trained for the face tampering task data, and an expert model is trained for the illegal image data.

[0053] Preferably, the second stage adopts the routing model for combination, and the result of the combination is used as the output of the hybrid expert model.

[0054] In the present invention, those skilled in the art may set the routing model based on experience, for example, using a ResNet model as the routing model.

[0055] In a preferred embodiment, the output of the hybrid expert model is expressed as:

[0056]

[0057] Among them, ∈ Mix represents the mixed expert model, t represents the time step, represents the input of the expert model at the tth time step, θ Mix represents the model parameters of the hybrid expert model, N represents the total number of expert models trained on normal and raw image pairing data, represents the jth expert model among the expert models trained on normal image and raw image pairing data, Represents an expert model The weight of Represents an expert model Model parameters; M represents the total number of expert models trained with backdoor paired data, represents the i-th expert model in the expert model trained with backdoor paired data, Represents an expert model The weight of Represents an expert model The model parameters.

[0058] More preferably, in the combination, the weights are also constrained, preferably by a regularization term, to balance the contributions of different expert models to the hybrid expert model, which can be expressed as:

[0059]

[0060] Among them, L whs represents the weight loss, Represents mean calculation, ‖‖ represents vector norm, and w is the weight matrix. and Get by combining.

[0061] In a preferred embodiment, the loss function for training the hybrid expert model is set to:

[0062]

[0063] Among them, p(j) represents the proportion of the j-th task data in the normal image pairing data to the training data set, p(i) represents the proportion of the i-th task data in the backdoor pairing data to the training data set, σ t is the preset noise parameter, and λ represents the degree to which the regularization term of the weight constraint affects the overall loss function;

[0064] |c means under normal image-raw image pairing data conditions, represents the generation process of the j-th normal graph at the t-th time step, Indicates generating a normal j-th image, Indicates that the input is a normal image of the jth category, Represents the expectation of generating normal j-th class image;

[0065] |p means under the backdoor paired data condition, represents the generation process of the t-th time step of the i-th type backdoor graph, Indicates the generation of backdoor i-th category image, Indicates that the input is the i-th type of backdoor image, Represents the expectation of generating backdoor i-th category image.

[0066] In S3, normal image generation: starting from time step t=T, a normal image is input to the trained hybrid expert model, and the output of the model is used as the input of the next time step t=T-1. The iteration continues until t=0 to generate the final result as a normal image.

[0067] Backdoor attack: Starting at time step t = T, an image with a trigger is input to the trained hybrid expert model. The model's output is used as the input for the next time step t = T-1. This is repeated until t = 0, generating a predefined backdoor image. In this invention, the model can be manipulated to generate heterogeneous backdoor images based on different triggers.

[0068] According to the method of the present invention, by performing a backdoor attack on the diffusion model, the stability and reliability of the diffusion model can be quickly tested, thereby evaluating the performance of the diffusion model.

[0069] Preferably, the method further comprises step S4, using the model obtained in step S3 as a post-attack model;

[0070] The expert model is trained using a dataset with only normal image and raw image pairing data to obtain the original diffusion model. The performance difference between the original diffusion model and the attacked model is obtained, and the stability of the expert model is evaluated based on the size of the difference.

[0071] Various embodiments of the methods described above in the present invention may be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which may be a special-purpose or general-purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0072] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0073] Example

[0074] Example 1

[0075] Conducting a backdoor attack experiment includes the following steps:

[0076] S1. Constructing a data set, wherein the data set includes backdoor pairing data, wherein the backdoor pairing refers to pairing an image with a trigger with a predefined backdoor image;

[0077] S2. Using the dataset to train a hybrid expert model, the hybrid expert model includes multiple expert models, each expert model is trained on a different generation task;

[0078] S3. Generate images using the trained model.

[0079] Among them, the dataset uses super-resolution task data as the normal image generation task, and one or more of the face tampering task data, illegal image task data, and illegal animation task data as the backdoor pairing data. The trigger in the backdoor pairing is set to the black square symbol in the lower right corner of the image. The backdoor comparison is expressed as:

[0080]

[0081] In S2, the training of the hybrid expert model includes two stages: the first stage is to train different expert models separately, and the second stage is to combine the trained expert models. The expert model is set to a diffuse Schrödinger bridge model.

[0082] In the second stage, the ResNet model is used as the routing model for combination. In the combination, the regularization term is used to constrain the weight redistribution and balance the contribution of different expert models to the hybrid expert model, which is expressed as:

[0083]

[0084] The output of the hybrid expert model is expressed as:

[0085]

[0086] The loss function for training the mixture of experts model is set as:

[0087]

[0088] Example 2

[0089] The same experiment as in Example 1 is performed, except that in S2, no weight redistribution process is performed in the second stage.

[0090] Example 3

[0091] The same experiment as in Example 1 was conducted, except that the expert model was performed using two UNet models with different parameter sizes. After step S3, there was a further step S4 in which the model obtained in step S3 was used as the post-attack model.

[0092] The expert model is trained using a dataset with only normal image and raw image pairing data to obtain the original diffusion model. The performance difference between the original diffusion model and the attacked model is obtained, and the stability of the expert model is evaluated based on the size of the difference.

[0093] Comparative Example 1

[0094] The same data set as in Example 1 is used to directly train the diffusion Schrödinger bridge model, and the trained model is used for image generation.

[0095] Experimental Example 1

[0096] The results of Example 1, Example 2 and Comparative Example 1 are shown in Table 1.

[0097] Table 1

[0098]

[0099]

[0100] FID (Fréchet Inception Distance) is a metric used to evaluate the quality of images generated by generative models. It quantifies the distance between the generated image and the real image by comparing their distribution differences in feature space. The lower the FID value, the more similar the generated image is to the real image.

[0101] PSNR stands for "Peak Signal-to-Noise Ratio", which is one of the indicators to measure image quality. The larger the PSNR value, the better the image quality.

[0102] SSIM (structural similarity index) is an indicator that measures the similarity between two images;

[0103] MSE (Mean Squared Error) is a commonly used statistical measure to evaluate the accuracy of a forecasting model or estimation method. It measures the error by calculating the average of the squares of the differences between the predicted and actual values.

[0104] CLIP score is an indicator used to evaluate the correlation between the image generated by the model and the original image;

[0105] ASR (Attack Success Rate) indicates the success rate of backdoor attacks, which is obtained by dividing the number of successful attacks by the total number of attacks;

[0106] Entropy is information entropy, which describes the similarity between the weights of different expert models. The larger the information entropy, the closer the weights of different expert models are, and thus the experts in backdoor attacks can be more covert.

[0107] As can be seen from Table 1, the method in Example 1 can generate high-quality images in normal generation tasks and accurately generate different backdoor images in heterogeneous backdoor attack tasks. Its performance is significantly better than that of Comparative Example 1. Compared with Example 2, Example 1 makes the routing weights closer to an even distribution through weight redistribution, thereby balancing the contribution of each expert to the final output of the model and making the model more stable.

[0108] Experimental Example 2

[0109] The results in Example 3 are statistically analyzed and shown in Tables 2 and 3.

[0110] Table 2 Attack performance of UNet models with different parameters

[0111]

[0112] Table III Performance changes after attack under different parameter UNet models

[0113]

[0114] From Table 2 and Table 3 we can see that:

[0115] (1) The UNet model with a larger number of parameters has better performance on normal tasks and its ability to generate malicious and illegal content after being attacked by backdoors than the model with a smaller number of parameters;

[0116] (2) By comparing the performance of the model before and after the backdoor attack on the normal image denoising task, it can be found that the backdoor attack affects the performance of the model on the normal task, but the model with a larger number of parameters is less affected by the backdoor attack, and the normal image denoising task index decreases relatively less;

[0117] The above demonstrates that a UNet model with a larger number of parameters has greater capacity, meaning it has greater potential and stronger performance. Therefore, under the influence of backdoor attacks, it is both more robust and stable on normal tasks, and more adaptable to new backdoor attack tasks. This is consistent with actual conditions, demonstrating that the method in Example 3 can quickly detect the stability and reliability of the diffusion model and quantitatively evaluate the stability performance of the diffusion model.

[0118] The present invention has been described above with reference to preferred embodiments, but these embodiments are merely exemplary and serve only as illustrations. On this basis, various replacements and improvements can be made to the present invention, all of which fall within the scope of protection of the present invention.

Claims

1. A backdoor attack method for a diffusion bridge model, characterized in that: The following steps are involved: S1. Constructing a data set, wherein the data set includes backdoor pairing data, wherein the backdoor pairing refers to pairing an image with a trigger with a predefined backdoor image; S2. Using the dataset to train a hybrid expert model, the hybrid expert model includes multiple expert models, each expert model is trained on a different generation task; S3. Generate images using the trained model.

2. The backdoor attack method for the diffusion bridge model according to claim 1 is characterized in that: In S1, the data set also includes normal image-generated image pairing data, where the normal image-generated image refers to a pairing of an image without a trigger and a generated image; In the backdoor pairing data, the trigger is a mark loaded on the normal image.

3. The backdoor attack method for the diffusion bridge model according to claim 1 is characterized in that: The backdoor pairing is represented as: Among them, δ represents the trigger, which is mixed into the normal input image Generated a backdoor image with a trigger M is a 0-1 matrix.

4. The backdoor attack method for the diffusion bridge model according to claim 1 is characterized in that: In S2, the training of the hybrid expert model includes two stages: the first stage is to train different expert models separately, and the second stage is to combine the trained expert models; The expert model is a diffusion model.

5. The backdoor attack method for the diffusion bridge model according to claim 4 is characterized in that: In the second stage, the routing model is used for combination, and the combined result is used as the output of the hybrid expert model.

6. The backdoor attack method for the diffusion bridge model according to claim 1 is characterized in that: The output of the hybrid expert model is expressed as: Among them, ∈ Mix represents the mixed expert model, t represents the time step, represents the input of the expert model at the tth time step, θ Mix represents the model parameters of the hybrid expert model, N represents the total number of expert models trained on normal and raw image pairing data, represents the jth expert model among the expert models trained on normal image and raw image pairing data, Represents an expert model The weight of Represents an expert model Model parameters; M represents the total number of expert models trained with backdoor paired data, represents the i-th expert model in the expert model trained with backdoor paired data, Represents an expert model The weight of Represents an expert model The model parameters.

7. The backdoor attack method for the diffusion bridge model according to claim 1 is characterized in that: In the combination, the weights are constrained to balance the contributions of different expert models to the hybrid expert model.

8. The backdoor attack method for the diffusion bridge model according to claim 1 is characterized in that: The method further comprises step S4, using the model obtained in step S3 as a post-attack model; The expert model is trained using a dataset with only normal image and raw image pairing data to obtain the original diffusion model. The performance difference between the original diffusion model and the attacked model is obtained, and the stability of the expert model is evaluated based on the size of the difference.

9. An electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-8.

Images