Image generation method of self-adaptive low-rank fine-tuning diffusion model based on differential privacy

Abstract

The invention relates to the technical field of image processing, and discloses a differential privacy-based image generation method of a self-adaptive low-rank fine-tuning diffusion model, which comprises the following steps of: acquiring a sensitive image data set and preprocessing the sensitive image data set; constructing a pre-trained diffusion model; encoding the preprocessed image into potential variable representation through a diffusion model, and adding Gaussian noise to the potential variable representation to form a noise image in the diffusion process; optimizing a module to be finely tuned in the diffusion model by adopting a self-adaptive low-rank fine tuning method; taking the noise image as the input of the back diffusion process, and applying differential privacy processing in the back diffusion training process to obtain a differential privacy adaptive low-rank fine-tuning diffusion model; and inputting the pure noise into the differential privacy self-adaptive low-rank fine-tuning diffusion model, executing a back diffusion process, and gradually denoising to obtain a generated image. By optimizing a specific module of the diffusion model and introducing a differential privacy mechanism, the quality of the model in image sample synthesis is improved on the premise of not sacrificing privacy protection.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more particularly to an image generation method based on an adaptive low-rank fine-tuning diffusion model based on differential privacy. Background Art

[0002] In recent years, with growing awareness of data privacy and technological advancements, using differential privacy (DP) to train generative models to produce privacy-preserving synthetic versions of sensitive data has become a research hotspot. In particular, methods based on diffusion models have attracted considerable attention due to their outstanding performance on image generation tasks. Diffusion models demonstrate their powerful capabilities in visual problems by gradually adding noise to the original image, mapping it to a standard normal distribution and then gradually restoring the clear image through a reverse process.

[0003] However, the limited privacy budget and expensive computational cost prevent the complex diffusion model from being fully learned, thus limiting the quality of the diffusion model on synthetic image samples.

[0004] Therefore, how to provide an image generation method based on an adaptive low-rank fine-tuned diffusion model with differential privacy is an urgent problem that needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides an image generation method based on an adaptive low-rank fine-tuned diffusion model with differential privacy. By optimizing specific modules of the diffusion model and introducing a differential privacy mechanism, the quality of the model on synthetic image samples is improved without sacrificing privacy protection, while reducing computational overhead, providing a more efficient and more privacy-protective image generation solution for sensitive data sets.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] An image generation method based on an adaptive low-rank fine-tuned diffusion model with differential privacy, including:

[0008] Obtain sensitive image datasets and perform preprocessing;

[0009] Build a pre-trained diffusion model;

[0010] The preprocessed image is encoded into a latent variable representation through a diffusion model, and Gaussian noise is added to it to form a noise image set in the diffusion process;

[0011] Adaptive low-rank fine-tuning method is used to optimize the diffusion model;

[0012] The noisy image set is used as the input of the back-diffusion process, and differential privacy processing is applied during the back-diffusion training process to obtain a differentially private adaptive low-rank fine-tuning diffusion model;

[0013] Pure noise is input into the differentially private adaptive low-rank fine-tuned diffusion model, and the reverse diffusion process is performed to gradually denoise and obtain the generated image.

[0014] Preferably, an adaptive low-rank fine-tuning method is used to optimize the diffusion model when fine-tuning the diffusion model, specifically including:

[0015] The weight update corresponding to each layer of the diffusion model is modeled as a weight increment matrix Δ, Δ={A l Λ l B l},in, and Represent the left and right singular vectors of Δ, the diagonal matrix It is initialized to a zero matrix, r<<min{d,p}, r is the rank in the weight increment matrix; and the original weight Keep frozen, d and p are matrices parameter;

[0016] Given input B is the batch size, M is the feature dimension, and the forward propagation is corrected by the weight increment matrix;

[0017] Calculate Λ using forward propagation l The diagonal elements of , calculate the importance score of each singular value;

[0018] According to the budget schedule, the singular values ​​are pruned and the rank r of the weight increment matrix is ​​dynamically adjusted.

[0019] Preferably, the forward propagation calculation formula is:

[0020]

[0021] Among them, φ is the activation function without trainable parameters, a l+1 is the output of this layer, b l is the model bias parameter, l∈L,L-1,…,1, and L represents the number of diffusion model layers.

[0022] Preferably, the noise image is used as the input of the back-diffusion process, and differential privacy processing is applied during the back-diffusion training process to obtain a differentially private adaptive low-rank fine-tuning diffusion model, specifically including:

[0023] Calculate the loss function based on the forward output result and the true label;

[0024] Calculate the gradient of each noise image based on the loss function;

[0025] Perform norm clipping on the gradient of each noise image;

[0026] Adding Gaussian noise to the clipped gradient to obtain a gradient with differential privacy;

[0027] The parameters of the low-rank increment matrix and the bias parameters of the diffusion model are updated through gradients with differential privacy until the privacy budget is exhausted, thereby obtaining an adaptive low-rank fine-tuning diffusion model with differential privacy.

[0028] Preferably, the gradient of each noise image is calculated:

[0029]

[0030] in, Indicates B l The gradient of the b-th sample is: Indicates A l The gradient of the b-th sample is: Represents Λ l The gradient of the b-th sample is: Denotes the gradient of the bias parameter with respect to the b-th sample, B l Trans Indicates B l The transpose of Λ l Trans Represents Λ l The transpose of is the loss function, Represents the output value of the activation function of the lth layer for the bth sample, Represents the activation function input value of the lth layer for the bth sample, is the loss function.

[0031] Preferably, the gradient calculation formula after clipping is:

[0032]

[0033] Where C is the clipping threshold, Represents the loss function For the l-th layer weight increment matrix Λ l The partial derivative of the i-th sample, Represents the loss function For the bias parameter b of the lth layer l The partial derivative of the i-th sample, Represents the loss function For the l-th layer weight increment matrix Λ l The clipped gradient of the i-th sample, Represents the loss function For the bias parameter b of the lth layer l The clipped gradient of the i-th sample.

[0034] Preferably, the gradient calculation formula with differential privacy is:

[0035]

[0036] in, Indicates that the mean is 0 and the covariance matrix is ​​σ 2 C 2 Gaussian noise of I, B t is the batch size used in the tth iteration, i.e. the sampled noise image, Represents the loss function For the l-th layer weight increment matrix Λ l The noise gradient of the i-th sample, that is, the gradient value that satisfies differential privacy, Represents the loss function For the bias parameter b of the lth layer l The noise gradient of the i-th sample, that is, the gradient value that satisfies differential privacy.

[0037] The above technical solution demonstrates that, compared to existing technologies, this invention provides an image generation method using an adaptive low-rank fine-tuning diffusion model based on differential privacy. By introducing partial fine-tuning techniques and an adaptive low-rank adjustment strategy, the generation quality of the diffusion model on sensitive image datasets is significantly improved while preserving differential privacy. The following are the specific contributions of this method to image generation:

[0038] 1) Improving the quality of synthesized images: By updating only the important parameters (i.e., singular values) in the weight increment matrix and applying differential privacy mechanisms to them, DP-AdaLoRA (adaptive low-rank fine-tuning method based on differential privacy) is able to retain more original information without sacrificing privacy protection. This makes the generated images not only have higher visual quality, but also more realistic and diverse.

[0039] 2) Enhanced model learning ability: DP-AdaLoRA uses a singular value decomposition (SVD)-based approach to optimize the incremental weight update process. This strategy allows the model to more effectively learn key features of the data distribution and maintain good performance even under limited privacy budgets.

[0040] 3) Reduced computational cost and memory consumption: Compared to traditional full fine-tuning methods, DP-AdaLoRA only requires training a very small percentage of parameters, significantly reducing computational and memory requirements. This means that even with large datasets or complex model structures, efficient training and inference processes can be achieved while ensuring the quality of generated images.

[0041] 4) Optimizing the balance between privacy protection and model performance: Through a carefully designed differential privacy mechanism, DP-AdaLoRA ensures that the gradient corresponding to each training sample is properly clipped and noise added, thereby strictly controlling the risk of privacy leakage.

[0042] In summary, DP-AdaLoRA, through its innovative partial fine-tuning technique and differential privacy mechanism, has achieved significant progress in the field of image generation. It not only effectively addresses the privacy limitations of existing diffusion models, but also significantly improves the quality and diversity of generated images, providing a new solution for the secure processing of sensitive datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0044] Figure 1 Flowchart of the image generation method based on the adaptive low-rank fine-tuning diffusion model based on differential privacy provided by the present invention.

[0045] Figure 2 A graph showing the change in regularization loss during model training provided by the present invention.

[0046] Figure 3 The present invention provides the rank distribution result of the weight increment matrix based on the method of the present invention.

[0047] Figure 4 The present invention provides a random sample of the CIFAR-10 dataset generated based on the method of the present invention. DETAILED DESCRIPTION

[0048] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0049] The embodiment of the present invention discloses an image generation method based on an adaptive low-rank fine-tuning diffusion model of differential privacy, such as Figure 1 Shown, including:

[0050] Obtain a sensitive image dataset D and preprocess it. The image data can be grayscale or color images, represented as a pixel matrix. Preprocess the dataset, including image normalization, resizing, and noise removal, to construct training data for fine-tuning the diffusion model.

[0051] Build a pre-trained diffusion model, including encoder, denoising network and decoder modules;

[0052] The preprocessed image is encoded into a latent variable representation through a diffusion model, and Gaussian noise is added to it to form a noise image set in the diffusion process;

[0053] Adaptive low-rank fine-tuning method is used to optimize the modules to be fine-tuned in the diffusion model;

[0054] The noisy image set is used as the input of the back-diffusion process, and differential privacy processing is applied during the back-diffusion training process to obtain a differentially private adaptive low-rank fine-tuning diffusion model;

[0055] Pure noise is input into the differentially private adaptive low-rank fine-tuned diffusion model, and the reverse diffusion process is performed to gradually denoise and obtain the generated image.

[0056] This method adds noise to key parameters, reducing computational and memory overhead while concentrating the allocation of the privacy budget. After trimming unimportant parameters, the privacy budget can be more concentratedly allocated to key parameters, allowing the privacy noise to act more concentratedly on the subspace with the largest amount of information, which helps the model learn more knowledge with the same privacy loss.

[0057] The adaptive low-rank fine-tuning and differential privacy training parts of the present invention are described in detail below.

[0058] The modules to be fine-tuned in the diffusion model are optimized using an adaptive low-rank fine-tuning method, specifically including:

[0059] (1) Insert low-rank incremental matrix:

[0060] Insert the weight increment matrix Δ into the linear layer (such as attention layer, convolution layer) of the diffusion model, Δ={A l Λ l B l},in, and Represent the left and right singular vectors of Δ, the diagonal matrix It is initialized to a zero matrix and in A l and B l A random Gaussian initialization is applied to ensure that Δ=0; and the original weights Keep frozen and only train B l , A l and Λl , the trainable parameter size is reduced from d×p to d×r+r×p+r, r<<min{d,p}, r is the rank in the weight increment matrix;

[0061] (2) Forward propagation process:

[0062] Given input B is the batch size, M is the feature dimension, and forward propagation is performed through the modified weights:

[0063]

[0064] Among them, φ is the activation function without trainable parameters, a l+1 is the output of this layer, l∈L,L-1,…,1, L represents the number of diffusion model layers;

[0065] Using the forward calculation l The diagonal elements (singular values) of , calculate the importance score of each singular value;

[0066] According to the budget timetable Retain parameters with high importance, cut off parameters with low importance, dynamically adjust the rank r of the weight increment matrix, optimize computing resources, and the budget schedule is used to control the overall budget during the training process.

[0067] In this way, the singular value directions that contribute most to the model performance can be retained first, and the directions corresponding to smaller singular values ​​can be ignored or simplified, thereby achieving efficient utilization of resources and improving model performance.

[0068] (3) Backpropagation and differential privacy training:

[0069] Based on the output results and the true label, calculate the loss function During the training process, in order to avoid the heavy calculation of singular value decomposition, a regularization term is introduced into the loss function to force A l and B l Orthogonal,improves the stability of parameter updates.

[0070]

[0071] ||·|| F represents the Frobenius norm of the matrix, I represents the identity matrix, R(A l ,B l ) represents the regularization term.

[0072] Calculate the gradient of each noise image separately and

[0073]

[0074] in, Indicates B l The gradient of the b-th sample is: Indicates A l The gradient of the b-th sample is: Represents Λ l The gradient of the b-th sample is: Denotes the gradient of the bias parameter with respect to the b-th sample, B l Trans Indicates B l The transpose of Λ l Trans Represents Λ l The transpose of is the loss function, Represents the output value of the activation function of the lth layer for the bth sample, It represents the activation function input value of the lth layer for the bth sample, and L is the loss function.

[0075] Here, It satisfies differential privacy. The generation of Λ l , so only and Applying differential privacy, the post-processing properties of differential privacy show that the weight increments still meet the same privacy bounds. This revised privacy protection significantly reduces the amount of noise introduced during each iteration. This reduction in noise, while maintaining the overall privacy lower bound, allows the model to retain more information, thereby learning more knowledge and improving model usability. This also reduces computational and memory overhead.

[0076] Clip the norm of the gradient of each noise image:

[0077]

[0078] Where C is the clipping threshold, Represents the loss function For the l-th layer weight increment matrix Λ l The partial derivative of the i-th sample, Represents the loss function For the bias parameter b of the lth layer l Partial derivative of the i-th sample;

[0079] Adding Gaussian noise to the clipped gradient, where the noise scale is determined by (∈,δ), satisfies (ε,δ)-differential privacy and obtains a gradient with differential privacy:

[0080]

[0081] in, Indicates that the mean is 0 and the covariance matrix is ​​σ 2 C 2 Gaussian noise of I, B t is the batch size used in the tth iteration, i.e. the sampled noise image;

[0082] The parameters of the weight increment matrix and the bias parameters of the diffusion model are updated through the gradient with differential privacy, and the original weight W l 0 The model does not participate in updates until the privacy budget is exhausted or the total number of iterations is reached, resulting in an adaptive low-rank fine-tuning diffusion model with differential privacy. The privacy budget is expressed as (ε, δ)-differential privacy, where ε (epsilon) measures the upper limit of privacy loss and δ (delta) represents the allowed failure probability. During training, each data access (such as gradient calculation) consumes a certain privacy budget, and the accumulated total budget must not exceed a pre-set threshold to ensure overall privacy protection.

[0083] To verify the effectiveness of the proposed method in the differential privacy diffusion model, an experimental environment was built based on guided-diffusion and compared with various efficient parameter fine-tuning methods. Specifically, the experimental settings are as follows:

[0084] Model architecture: Contains 2 residual blocks per resolution, has a base number of 192 channels, uses 1 attention head (16 channels per head), introduces the attention mechanism at 16x16 resolution, and has a channel multiplication factor of (1, 2, 2, 2).

[0085] Diffusion process: 1000-step linear noise scheduling is used.

[0086] Training strategy: Apply enhanced diversity and time-step diversity to improve sample diversity without increasing the privacy budget.

[0087] Hardware configuration: All experiments are run on an 80GB A800 GPU.

[0088] Main experimental results:

[0089] The performance of DP-AdaLoRA is compared with other parameter-efficient fine-tuning methods on the CIFAR-10 dataset, including full fine-tuning and DP-BiTFiT. Tables 1-4 show the comparison of the generation results of different differential privacy fine-tuning diffusion model methods under different privacy budgets ∈ on the CIFAR-10 dataset:

[0090] Table 1

[0091]

[0092] Table 2

[0093]

[0094] Table 3

[0095]

[0096] Table 4

[0097]

[0098] Results show that both differentially private biased fine-tuning (DP-BiTFiT) and our proposed differentially private adaptive low-rank fine-tuning (DP-AdaLoRA) outperform full fine-tuning (DP-Full). Furthermore, our method achieves better results than DP-BiTFiT, especially when the privacy budget is 5, 10, and 50.

[0099] To further explain why AdaLoRA cannot be directly applied to differential privacy training, we visualize the orthogonality loss of the singular vectors, as shown in Figure 2 As shown. Figure 2 As can be seen in the figure, when the adaptive low-rank fine-tuning method is directly applied, the loss value remains almost unchanged during the training process. However, after applying the optimization strategy, that is, the adaptive low-rank fine-tuning diffusion model method based on differential privacy that we proposed, the loss gradually decreases to 0. This shows that adding noise to singular values ​​and singular vectors at the same time, and the subsequent multiplication operation between the three, can even amplify the impact of the noise, ultimately severely reducing the usability of the model.

[0100] We also visualize the rank of the low-rank delta matrix at each layer. Figure 3 As shown in the figure, the horizontal axis represents the number of model layers, which increases from top to bottom and from left to right. The vertical axis represents the rank of the increment matrix. The results show that at the end of training, the rank of the weight increments in each layer of the model varies. This is because the importance scores in the fine-tuning technique focus the method on key modules. This means that the proposed method can focus more resources on the calculation of important weights, saving unnecessary resource consumption.

[0101] like Figure 4 As shown, it is shown that the method of the present invention can achieve high-quality and high-efficiency image generation capabilities.

[0102] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0103] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image generation method based on an adaptive low-rank fine-tuned diffusion model with differential privacy, characterized in that: include: Obtain sensitive image datasets and perform preprocessing; Build a pre-trained diffusion model; The preprocessed image is encoded into a latent variable representation through a diffusion model, and Gaussian noise is added to it to form a noise image set in the diffusion process; Adaptive low-rank fine-tuning method is used to optimize the diffusion model; The noisy image set is used as the input of the back-diffusion process, and differential privacy processing is applied during the back-diffusion training process to obtain a differentially private adaptive low-rank fine-tuning diffusion model; Pure noise is input into the differentially private adaptive low-rank fine-tuned diffusion model, and the reverse diffusion process is performed to gradually denoise and obtain the generated image.

2. The image generation method based on the adaptive low-rank fine-tuning diffusion model of differential privacy according to claim 1 is characterized in that: When fine-tuning the diffusion model, an adaptive low-rank fine-tuning method is used for optimization, including: The weight update corresponding to each layer of the diffusion model is modeled as a weight increment matrix Δ, Δ={A l Λ l B l },in, and Represent the left and right singular vectors of Δ, the diagonal matrix It is initialized to a zero matrix, r<<min{d,p}, r is the rank in the weight increment matrix; and the original weight Keep frozen, d and p are matrices parameter; Given input B is the batch size, M is the feature dimension, and the forward propagation is corrected by the weight increment matrix; Calculate Λ using forward propagation l The diagonal elements of , calculate the importance score of each singular value; According to the budget schedule, the singular values ​​are pruned and the rank r of the weight increment matrix is ​​dynamically adjusted.

3. The image generation method based on the adaptive low-rank fine-tuning diffusion model of differential privacy according to claim 2 is characterized in that: The forward propagation calculation formula is: Among them, φ is the activation function without trainable parameters, a l+1 is the output of this layer, b l is the model bias parameter, l∈L,L-1,…,1, and L represents the number of diffusion model layers.

4. The image generation method based on the adaptive low-rank fine-tuning diffusion model of differential privacy according to claim 1, characterized in that: The noise image is used as the input of the back-diffusion process, and differential privacy processing is applied during the back-diffusion training process to obtain a differentially private adaptive low-rank fine-tuning diffusion model. Specifically, it includes: Calculate the loss function based on the forward output result and the true label; Calculate the gradient of each noise image based on the loss function; Perform norm clipping on the gradient of each noise image; Adding Gaussian noise to the clipped gradient to obtain a gradient with differential privacy; The parameters of the low-rank increment matrix and the bias parameters of the diffusion model are updated through gradients with differential privacy until the privacy budget is exhausted, thereby obtaining an adaptive low-rank fine-tuning diffusion model with differential privacy.

5. The image generation method based on the adaptive low-rank fine-tuning diffusion model of differential privacy according to claim 4 is characterized in that: Compute the gradient of each noisy image: in, Indicates B l The gradient of the b-th sample is: Indicates A l The gradient of the b-th sample is: Represents Λ l The gradient of the b-th sample is: Denotes the gradient of the bias parameter with respect to the b-th sample, B l Trans Indicates B l The transpose of Λ l Trans Represents Λ l The transpose of is the loss function, a l (b) Represents the output value of the activation function of the lth layer for the bth sample, y l (b) Represents the activation function input value of the lth layer for the bth sample, is the loss function.

6. The image generation method based on the adaptive low-rank fine-tuning diffusion model of differential privacy according to claim 4, characterized in that: The gradient calculation formula after clipping is: Where C is the clipping threshold, Represents the loss function For the l-th layer weight increment matrix Λ l The partial derivative of the i-th sample, Represents the loss function For the bias parameter b of the lth layer l The partial derivative of the i-th sample, Represents the loss function For the l-th layer weight increment matrix Λ l The clipped gradient of the i-th sample, Represents the loss function For the bias parameter b of the lth layer l The clipped gradient of the i-th sample.

7. The image generation method based on the adaptive low-rank fine-tuning diffusion model of differential privacy according to claim 6, characterized in that: The gradient calculation formula with differential privacy is: in, Indicates that the mean is 0 and the covariance matrix is ​​σ 2 C 2 Gaussian noise of I, B t is the batch size used in the tth iteration, i.e. the sampled noise image, Represents the loss function For the l-th layer weight increment matrix Λ l The noise gradient of the i-th sample, that is, the gradient value that satisfies differential privacy, Represents the loss function For the bias parameter b of the lth layer l The noise gradient of the i-th sample, that is, the gradient value that satisfies differential privacy.

Images