Diffusion model LoRA fine tuning optimization method and system based on CLIP loss and perception loss

Abstract

The invention provides a diffusion model LoRA fine tuning optimization method and system based on CLIP loss and perception loss, and the method comprises the steps: 1, dynamically adjusting the weights of the CLIP loss and the perception loss in combination with the CLIP loss and the perception loss in a LoRA fine tuning process; 2, calculating the semantic similarity between the denoised intermediate image and the target text by using a CLIP model, and optimizing the noise prediction capability of a diffusion model according to the similarity difference; step 3, calculating the difference between the intermediate image and the target image in the feature space by using perception loss, optimizing the noise prediction capability of the diffusion model, and improving the visual quality and detail fidelity of the generated image; and step 4, adjusting whether to start CLIP loss and perception loss or not according to the training progress. By introducing the CLIP loss, the model can better align the image and the text in the fine tuning training process, so that the generated image better conforms to the description of the text prompt.

Description

Technical Field

[0001] The present invention relates to the technical field of deep neural network, and in particular to a diffusion model LoRA fine-tuning optimization method and system based on CLIP loss and perceptual loss. Background Art

[0002] The diffusion model is a generative model based on probability distribution. Its basic idea is to gradually add noise to real data so that it eventually approaches the standard Gaussian distribution, which is called forward diffusion. Then, a model is trained to learn the reverse process, and real data is gradually restored from random noise to achieve data generation. In the forward diffusion process, the model injects noise into the data in fixed steps, gradually destroying its structure until the data completely becomes unstructured Gaussian noise. The goal of reverse generation is to gradually remove noise and restore realistic data distribution. At this stage, the requirements of a large number of users and industrial needs for AIGC (artificial intelligence generated content) have become more segmented. Different fields and individual users need to generate content that can more accurately meet specific scene requirements. This trend has promoted the birth and development of fine-tuning technology. Fine-tuning technology enables the model to adapt to specific tasks or user needs while maintaining its original capabilities by efficiently adjusting the pre-trained model, thereby achieving a more personalized and customized generation effect.

[0003] Fine-tuning technology is a very important research direction in the field of diffusion models. The core idea of ​​fine-tuning technology is to use a pre-trained large model as the basis, and through a small amount of new task data and targeted training, make the model focus on a specific field or task. Traditionally, the training of deep learning models usually requires a large amount of data and computing resources, but fine-tuning technology can take advantage of the wide adaptability of pre-trained models to significantly reduce the training cost of new tasks while avoiding building models from scratch. Common fine-tuning methods include full parameter fine-tuning and partial parameter fine-tuning, such as freezing some parameters and optimizing only specific layers, but most fine-tuning technologies still face problems such as high computing resource requirements and large number of model parameters saved after training. Because fine-tuning technology usually relies on large-scale pre-trained models, and the training and optimization of these models often require powerful computing resources, such as high-performance GPU clusters and large-capacity storage devices, it makes it difficult for most ordinary users and small teams to bear the corresponding costs and technical complexity. Even though fine-tuning has significantly reduced resource requirements compared to training models from scratch, for many users, they still need to be familiar with deep learning frameworks, adjust hyperparameters, and prepare high-quality domain data. These barriers hinder the popularization of fine-tuning technology.

[0004] To solve this problem, the LoRA fine-tuning technology has emerged in recent years. LoRA (Low-Rank Adaptation) is a lightweight model fine-tuning technology that aims to reduce the computational cost and storage requirements of fine-tuning on large pre-trained models while maintaining the flexibility and effect of fine-tuning. The core idea of ​​LoRA is to use low-rank matrix decomposition to efficiently update model parameters instead of adjusting the weights of the entire pre-trained model. During the fine-tuning process, LoRA freezes all the original parameters of the pre-trained model and inserts trainable low-rank matrices in the specified network layer. These low-rank matrices are used to capture the specific task information newly added during the fine-tuning process, thereby achieving efficient fine-tuning while reducing the number of parameter updates. Summary of the invention

[0005] Purpose of the invention: This invention proposes a LoRA (Low-Rank Adaptation) fine-tuning optimization method for the diffusion model based on CLIP loss and perceptual loss. This method significantly improves the overall quality of the generated image by dynamically optimizing the performance of semantic consistency and visual details. Specifically, it includes the following steps:

[0006] Step 1: In the LoRA fine-tuning process, CLIP loss and perceptual loss are combined to ensure the comprehensive performance of the generated image in terms of semantic consistency and visual details by dynamically adjusting the weights of CLIP loss and perceptual loss.

[0007] Step 2: randomly sample time step t, use the CLIP model to calculate the semantic similarity between the denoised intermediate image and the target text, and optimize the noise prediction ability of the diffusion model based on the similarity difference;

[0008] Step 3: Use perceptual loss to calculate the difference between the intermediate image and the target image in the feature space, and optimize the noise prediction ability of the diffusion model to improve the visual quality and detail fidelity of the generated image;

[0009] Step 4: Adjust whether to enable CLIP loss and perceptual loss according to the training progress;

[0010] In step 1, a stable diffusion model is used as a pre-trained diffusion model. In order to speed up the training time, the stable diffusion model uses a VAE (Variational Autoencoder) model to compress the image into a latent space for calculation during the training phase. The latent space means reducing the image size but increasing the number of channels, in order to reduce the amount of calculation when calculating pixel by pixel. The VAE model includes an encoder and a decoder. The encoder is used to compress the image into the latent space, and the decoder is used to restore the image size.

[0011] During the LoRA fine-tuning process, in the initial fine-tuning training stage, the stable diffusion model is first allowed to focus on predicting noise. At this time, because the model noise prediction is not yet stable, adding the CLIP loss term too early can easily lead to a deterioration in the model prediction noise effect. The following formula is used to calculate the standard deviation of the conventional loss :

[0012]

[0013] Where k is the sliding window size, which is set to 10 by default; set the threshold θ, which is generally set to 0.07; is the average loss value in the sliding window; when the loss of the conventional stable diffusion model is within k steps, the standard deviation When it is less than or equal to the threshold θ, the conventional loss is judged to be stable, and the CLIP loss is enabled at this time; represents the current time step, t represents the time from arrive All time steps that can be sampled in the window; It represents the loss of the conventional stable diffusion model calculated under the current sampling step;

[0014] The conventional stable diffusion model loss is defined as:

[0015]

[0016] in represents the real noise, represents the prediction noise of the stable diffusion model; N represents the number of samples in a batch;

[0017] In the present invention, the loss calculation formulas for CLIP loss and perceptual loss are improved respectively, and under the loss formulas, CLIP loss and perceptual loss can adjust weights adaptively. In CLIP loss, the improvement can help CLIP loss prevent the instability of model training due to excessively large number of sampled noise steps, which leads to the difference between semantics and noisy images. Under perceptual loss, the improvement helps perceptual loss to automatically adjust the weights of each layer and better adjust the optimization parameters.

[0018] In step 2, after enabling the CLIP loss term, the pre-trained CLIP model is used to calculate the semantic similarity between the image and the target text. In the present invention, considering that the CLIP model is applied in the training stage, if the sampled time step t is too large in the training stage, the noisy image x t There is too much noise in the image. At this time, performing CLIP calculation is likely to cause unstable loss. Therefore, in the present invention, a loss function is designed for the CLIP model to adapt to noisy images of different degrees in the stable diffusion model. The loss function formula is defined as:

[0019]

[0020]

[0021] Where T represents the maximum time step in the stable diffusion model, which defaults to 1000; Represents the vector value after CLIP encoding of the image; Represents the vector value after encoding the text; represents the CLIP loss term; Represents the weight value, calculated based on the time step t;

[0022] Step 2.1: In order to use perceptual loss, the stable diffusion model denoising process is modified. First, the target fine-tuning image X0 is obtained from the target fine-tuning dataset, and then the fine-tuning image X0 is compressed using the encoder in the VAE model to obtain the compressed target fine-tuning image. , and then fine-tune the compressed target image Perform noise addition t-1 steps to obtain the noise image x t-1 This step is to provide the real x for the subsequent calculation of perceptual loss t-1 Image, and then the current noise image x t-1 Add one more step of noise to get the noisy image x t , while retaining the Gaussian noise randomly sampled at step t Used for MSE (Mean Squared Error Loss) to calculate conventional loss pixel by pixel;

[0023] For the current noise image x t Perform single-step denoising of the conventional stable diffusion model, specifically including: fine-tuning the compressed target image The corresponding text text is converted into a text vector T through CLIP encoding. text , then T text and the noisy image x t Input into the stable diffusion model, the stable diffusion model will combine the internal parameters of LoRA to predict the noise and obtain the predicted noise , then use Remove the noise added in step t to obtain the predicted noisy image ;

[0024] Step 2.2, for the predicted noisy image Perform CLIP image encoding to obtain image vector I;

[0025] Step 2.3, calculate the text vector T textThe semantic similarity with the image vector I. According to step 2.1, in the process of predicting noise, LoRA participates in the calculation of the stable diffusion model, so this part of the calculation will be passed to LoRA during back propagation. During the back propagation process, the gradient descent algorithm will be calculated according to the conventional loss term and the CLIP loss term in the present invention, and finally used to adjust the parameters in LoRA.

[0026] Step 2.1 includes:

[0027] Step 2.1.1, randomly sample one step of Gaussian noise , Gaussian noise Used for the t-1th step of noise addition, and then use the stable diffusion model to add noise formula x t-1 , and get the noise added The noisy target image x t-1 ;

[0028] Among them, in the stable diffusion model, a set of parameters are predefined through linear data sampling scheduling , Refers to a set of parameters The tth value in refers to ;

[0029] Step 2.1.2, obtain random Gaussian noise For the current noisy target image x that has been noisy for t-1 steps t-1 Perform another single-step noise addition, the noise addition formula is:

[0030] ,

[0031] Get the image after adding noise , while preserving the Gaussian noise As real noise; Refers to a set of parameters predefined by linear data sampling scheduling in a stable diffusion model The tth data in ;

[0032] Step 2.1.3: Use the CLIP model to perform word embedding on the text. First, split the text into independent words or subword units, and then embed the units through the text encoder of the CLIP model to generate a text vector T corresponding to the text semantics. text ; Text vector T text It can capture the semantic information of text and serve as a reference for image generation in the multimodal alignment process;

[0033] Step 2.1.4, transform the text vector T text, the image after adding noise and the current time step t are passed into the stable diffusion model combined with LoRA, allowing the model to predict the amount of noise added at step t , for the image Remove Step noise, use the following formula to get the noisy image:

[0034] .

[0035] Step 2.2 includes:

[0036] Step 2.2.1, considering the application limitations of the CLIP model, after obtaining a noisy image After that, the noisy image is first converted to The decoder of the VAE model restores the size to obtain the original size image with noise ,The purpose of this step is to match the input requirements of the CLIP model;

[0037] Step 2.2.2: Use CLIP’s image encoder to decode the noisy original size image. Extract feature vectors and convert the original noisy image into Converted into an image vector I, the image vector I captures the key semantic information of the image and can be used to measure the similarity between the image and the text description, thereby achieving semantic alignment or optimization in multimodal tasks.

[0038] Step 2.3 includes:

[0039] Step 2.3.1, use the loss formula For the text vector T text And the image vector I is used for loss calculation, where Replaced by T, By replacing I, we get As the semantic similarity between the text vector and the image vector I;

[0040] Step 2.3.2, determine the weight setting at this time according to the current time step t , considering that the calculation of CLIP loss is meaningless when t is large, setting a large weight at this time often leads to the model's prediction of loss out of control. , adjust the weight value, when t is large, it can retain a larger weight value, and when t is small, adjust the weight to a suitable value;

[0041] Step 2.3.3, since the present invention uses the original size image with noise when calculating the CLIP loss , the LoRA bypass parameter in the denoising process directly participates in the noise Therefore, the CLIP loss is introduced into the gradient calculation graph, and the calculated CLIP loss gradient is passed back along the optimization path to adjust the parameters in LoRA.

[0042] Step 3 includes:

[0043] Step 3.1, considering the disadvantage of traditional perceptual loss, it cannot perceive the importance of losses at different layers. In traditional perceptual loss, a deep convolutional network is first used to extract features from the image, and features are extracted at different levels, and finally a simple sum is used as the total loss. In the present invention, a new perceptual loss function is used to optimize the network results, and the formula is:

[0044]

[0045]

[0046] in represents the total loss value of the perceived loss, Indicates the feature extraction of the image by each layer of convolutional network. represents a dynamic weight term, which will adjust the final total perceptual loss composition according to the loss values ​​of different layers; exp is a natural exponential function; represents the feature extraction network of the kth layer;

[0047] Step 3.2, considering the design of perceptual loss, the method of step 2.2.1 is used to distort the noisy image After resizing, perceptual loss is used for calculation to ensure the feature integrity of subsequent inputs;

[0048] Step 3.3, using perceptual loss For the original noisy image The perceptual loss is calculated by calculating the original size image with noise The difference values ​​between the target fine-tuned image X0 and the various levels in the perception model are used to better guide the adjustment of the internal parameters of LoRA during back propagation. Because of the adaptive loss function of the present invention, the perceptual loss will back propagate and adjust the parameters in LoRA according to the importance of each layer, so that the model can more finely capture the details and semantic differences beyond the pixel level, so as to improve the visual quality and detail fidelity of the generated image.

[0049] Step 3.3 specifically includes:

[0050] Step 3.3.1, layer the feature extraction network VGG16, select the 0th, 1st, 2nd, 3rd, 4th, and 5th convolution outputs as the five feature vectors of the image; add a linear layer to the output features and perform linear changes. Here, neurons are killed with a probability of 20% to prevent overfitting; the input channels of the feature extraction network VGG16 are adjusted to 64, 128, 256, 512, 512 to better capture the feature information of multi-channel images.

[0051] Step 3.3.2: Convert the original size noisy target image The target fine-tuned image X0 is input into the feature extraction network VGG16. During the calculation process, the feature extraction network VGG16 saves the calculation results in the 0th, 1st, 2nd, 3rd, 4th, and 5th layers for calculating the loss.

[0052] Step 3.3.3, calculate the original size noisy target image And the second-order norm loss of the target fine-tuning image X0 after passing through different feature extraction layers , and record the loss value. This step is used to calculate the size of the weighted value. The formula is:

[0053] Calculate the weighted value of each layer and save it, and finally use the formula to sum the total loss:

[0054] .

[0055] in represents the perceptual loss term.

[0056] Step 4 includes: defining the following final loss function:

[0057]

[0058] in, refers to the total loss value, is the CLIP loss, For perceptual loss, when the CLIP loss enabling condition is not met, the parameter Set to 0; after CLIP loss is enabled, Set to 1.

[0059] Through back propagation, the total loss signal is passed to the LoRA model to optimize its parameter update. This process can effectively guide the model to capture details and semantic features, further improving the overall quality of generated images and text alignment effects.

[0060] The present invention also provides a diffusion model LORA fine-tuning optimization system based on CLIP loss and perception loss, comprising:

[0061] The CLIP loss calculation module is used to: encode the text corresponding to the target fine-tuning image to obtain the text vector, and then After adding noise for t steps, the stable diffusion model combined with LoRA is used to predict the noise of the tth step, and then the predicted noise is subtracted to obtain the noisy image The purpose is to introduce CLIP loss into the gradient calculation graph for back propagation adjustment of LoRA; then the noisy image Use the decoder of the VAE model to restore the size, then use the CLIP model of the present invention to encode the restored image into a vector, and finally use the adaptive weight loss calculation formula of the present invention to calculate the difference between the text vector and the image vector;

[0062] The perceptual loss calculation module is used to: firstly calculate the target fine-tuned image X0 and the noisy image from which the noise of the tth step has been removed The input feature extraction network VGG16, this model specifies to extract features on the 0th, 1st, 2nd, 3rd, 4th, and 5th convolution layers in VGG16. Finally, in order to allow the model to adaptively adjust the different importance of each layer of perceptual loss, the adaptive perceptual loss in the present invention is used to calculate the features extracted above.

[0063] Finally, after merging the CLIP loss and the perceptual loss with the base loss, backpropagation is performed to guide the parameter update inside LoRA.

[0064] Beneficial effects: The present invention integrates CLIP loss and perceptual loss into LoRA fine-tuning, thereby enhancing the effect of LoRA fine-tuning and reducing the number of times and pictures required for fine-tuning training. By introducing CLIP loss, the model can better align images with text during fine-tuning training, making the generated images more consistent with the description of the text prompt. The introduction of perceptual loss enables the model to learn to align the generated image with the target image during training, thereby improving the quality and visual effect of the generated image. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0066] Figure 1 This is a flow chart of adding noise to the model in the present invention.

[0067] Figure 2 This is a flow chart of the noise prediction using the model in the present invention combined with LoRA.

[0068] Figure 3 This is the calculation process of the two additional loss terms proposed in the present invention.

[0069] Figure 4 A comparison chart of the effects of the present invention, the original LoRA and SD-2.1V is shown. DETAILED DESCRIPTION

[0070] The embodiment of the present invention provides a diffusion model LoRA fine-tuning optimization method based on CLIP loss and perception loss. Figure 1 The model's denoising process is shown, which starts with obtaining a data image X from the target fine-tuning image. First, the target image X is input into the encoder of the variational autoencoder (VAE) for processing. In the present invention, the original image X is a 512×512 image. When scaling the image, the experiment chooses a strategy of scaling it to 8 times the original size. The experimental results show that when the scaling ratio is between 8 and 16 times, it can not only well preserve the image details and avoid a large amount of detail loss during decoding, but also effectively reduce the consumption of computation.

[0071] After being processed by the encoder, the image is compressed into a latent feature matrix Z, which is a highly concentrated latent representation for subsequent noise addition and loss calculation. Next, the latent feature matrix Z is gradually noised, first to the t-1 step. During this process, the noise component of the t-1 step is gradually mixed into the latent feature matrix Z, and is used to calculate the perceptual loss in this state to ensure that the addition of noise does not significantly affect the representation ability of the feature.

[0072] Subsequently, Z, which already contains t-1 steps of noise, is subjected to a single-step noise addition to make it noisy to the current randomly selected time step t. In this noise addition process, the noise term ε is additionally retained for the calculation of subsequent conventional losses (such as L2 loss). In this way, the model can effectively balance the relationship between the preservation of image details and computational complexity while gradually adding noise, and lay the foundation for high-quality generation and reconstruction.

[0073] Figure 2 The overall process of model prediction of noise is shown, and how the latent feature vector Z and text information T can be used to efficiently predict noise through the U-Net network is explained in detail. In this process, the latent feature vector Z and text information T obtained from the previous stage are first input into the U-Net network for processing. The architecture of the U-Net network consists of two parts: the encoder and the decoder, and multiple cross-attention modules are embedded inside, which are specifically used to effectively fuse text and image information.

[0074] The first step in the processing is to feed the input into the encoder module. In the encoder, each layer contains a cross-attention mechanism, which completes the deep fusion of the two by calculating the attention weights between text and image features. In this process, the LoRA (Low-Rank Adaptation) parameters and the original parameters in the cross-attention module are merged to complete the calculation. However, the original parameters of the cross-attention module are frozen at this stage, which means that only the LoRA parameters will be optimized during backpropagation. This design not only reduces the number of parameters that need to be adjusted, but also significantly improves the training efficiency of the model while ensuring the stability of the core attention mechanism.

[0075] After being processed by the encoder, a compressed latent feature matrix that combines text information and image information is generated. This matrix not only retains the spatial representation of image features, but also contains the semantic information of text description, providing rich context for subsequent generation.

[0076] Next, the compressed latent feature matrix is ​​passed to the decoder module. In the decoder, the network restores the low-resolution feature matrix to an output that matches the original image resolution through a step-by-step upsampling operation. In this process, the decoder not only uses a cross-attention module, but also introduces a low-rank adaptation (LoRA) parameter to achieve a deep fusion of text and image information through these components. This design ensures that each step of the generation process fully utilizes the text guidance information, thereby enhancing the relevance of the generated results to the text description while retaining the image's detailed expression and layering, providing a double guarantee for the quality of the final generated image.

[0077] Figure 3 The calculation process of the two additional loss terms proposed in the present invention is shown. Specifically, the predicted latent feature vector Z containing t-1 step noise is first processed by the decoder to enlarge its size back to the resolution of the original image. At the same time, the original image with t-1 step noise is also restored to its original size through the decoder to meet the input requirements of the subsequent CLIP model and VGG16 network. In the specific process of calculating the loss, the CLIP loss is calculated first. The text information of the current original image is extracted through the CLIP model, and it is calculated with the latent feature Z after the size is restored to obtain the CLIP loss term, which is used to measure the matching degree between the generated content and the text description.

[0078] Next, the restored predicted image X and the latent feature Z are input into the VGG16 network to calculate the multi-layer perceptual loss. The multi-layer perceptual loss can more comprehensively capture the differences in details and styles of the generated images by comparing and analyzing the perceptual results of different feature layers, further improving the generated quality. Finally, the multi-layer perceptual loss is averaged to return a scalar loss term to guide model optimization.

[0079] After obtaining the CLIP loss, multi-layer perception loss, and conventional noise loss terms, these loss terms are combined using the weighted loss calculation formula to generate the final overall loss value. Specifically, by reasonably setting the weights of each loss term, ensure that their contribution to model training during the optimization process is balanced and effective. Subsequently, the parameters are optimized through the back-propagation algorithm. During the gradient descent process, the model automatically updates the internal parameters of LoRA, while the weights of the diffusion model are frozen and will not participate in the update calculation. This design greatly reduces the computational overhead required for training while effectively utilizing the high efficiency of LoRA.

[0080] In the present invention, the total number of parameters of the model is 872,550,340, but only 6,639,616 parameters are actually involved in the optimization calculation, accounting for less than 1%. This significant reduction in the number of parameters greatly reduces the computational burden and makes the training process more efficient. Taking an ordinary machine equipped with an RTX 4070 SUPER graphics card as an example, running 5000 training steps only takes about half an hour. This efficient training speed greatly improves the practicality of the model, and also provides the possibility of large-scale model training in resource-constrained hardware environments.

[0081] In addition, by freezing the weights of the diffusion model, its original performance is not affected, while the optimization of LoRA parameters can fully explore the potential of text-image matching. This targeted optimization strategy not only significantly improves the quality of generated images, but also reduces the dependence on hardware performance, providing a reliable foundation for the implementation and further improvement of the model.

[0082] Figure 4 The experimental results of the comparison between the LoRA fine-tuning method proposed in the present invention and the traditional LoRA fine-tuning method and the conventional diffusion model are shown. From the leftmost column to the rightmost column, they are: the method proposed in the present invention, the traditional LoRA fine-tuning method and the original stable diffusion model. The experiment used the rich woman dataset, which is characterized by vivid characters, including the diverse features of wearing bright clothes and accessories such as sunglasses. Under the same conditions of generating seeds and prompt texts, the optimized LoRA fine-tuning method of the present invention shows significant advantages. It can not only understand and follow the prompt text more accurately, but also effectively improve the quality and detail fidelity in the generated image.

[0083] The present invention provides a method and system for fine-tuning and optimizing a diffusion model LoRA based on CLIP loss and perceptual loss. There are many methods and ways to implement the technical solution. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention. All components not specified in this embodiment can be implemented using existing technologies.

Claims

1. A LoRA fine-tuning optimization method for a diffusion model based on CLIP loss and perceptual loss, characterized in that: The steps include: Step 1: During the LoRA fine-tuning process, the weights of CLIP loss and perceptual loss are dynamically adjusted by combining them. Step 2: randomly sample time step t, use the CLIP model to calculate the semantic similarity between the denoised intermediate image and the target text, and optimize the noise prediction ability of the diffusion model based on the similarity difference; Step 3: Use perceptual loss to calculate the difference between the intermediate image and the target image in the feature space and optimize the noise prediction ability of the diffusion model; Step 4: Adjust whether to enable CLIP loss and perceptual loss based on the training progress.

2. The method according to claim 1, characterized in that In step 1, a stable diffusion model is used as a pre-trained diffusion model, and a VAE model is used to compress the image into a latent space for calculation during the training phase; the VAE model includes an encoder and a decoder, the encoder is used to compress the image into the latent space, and the decoder is used to restore the image size; During the LoRA fine-tuning process, the following formula is used to calculate the standard deviation of the conventional loss: ， Where k is the sliding window size; is the average loss value in the sliding window; when the loss of the conventional stable diffusion model is within k steps, the standard deviation When it is less than or equal to the threshold θ, the conventional loss is judged to be stable, and the CLIP loss is enabled at this time; represents the current time step, t represents the time from arrive All time steps that can be sampled in the window; It represents the loss of the conventional stable diffusion model calculated under the current sampling step; The conventional stable diffusion model loss is defined as: ， in represents the real noise, represents the stable diffusion model prediction noise; N represents the number of samples in a batch.

3. The method according to claim 2, characterized in that In step 2, after enabling the CLIP loss term, the pre-trained CLIP model is used to calculate the semantic similarity between the image and the target text. A loss function is designed for the CLIP model to adapt to different degrees of noisy images in the stable diffusion model. The loss function formula is defined as: ， ， Where T represents the maximum time step in the stable diffusion model; Represents the vector value after CLIP encoding of the image; Represents the vector value after encoding the text; represents the CLIP loss term; Represents the weight value; Step 2.1: In order to use perceptual loss, the stable diffusion model denoising process is modified. First, the target fine-tuning image X0 is obtained from the target fine-tuning dataset, and then the fine-tuning image X0 is compressed using the encoder in the VAE model to obtain the compressed target fine-tuning image. , and then fine-tune the compressed target image Perform noise addition t-1 steps to obtain the noise image x t-1 , and then for the current noise image x t-1 Add one more step of noise to get the noisy image x t , while retaining the Gaussian noise randomly sampled at step t Used for MSE to calculate conventional loss pixel by pixel; For the current noise image x t Perform single-step denoising of the conventional stable diffusion model, specifically including: fine-tuning the compressed target image The corresponding text text is converted into a text vector T through CLIP encoding. text , then T text and the noisy image x t Input into the stable diffusion model, the stable diffusion model will combine the internal parameters of LoRA to predict the noise and obtain the predicted noise , then use Remove the noise added in step t to obtain the predicted noisy image ; Step 2.2, for the predicted noisy image Perform CLIP image encoding to obtain image vector I; Step 2.3, calculate the text vector T text The semantic similarity between the image vector I and the image vector I is calculated by the gradient descent algorithm according to the conventional loss term and the CLIP loss term during the back propagation process.

4. The method according to claim 3, characterized in that Step 2.1 includes: Step 2.1.1, randomly sample one step of Gaussian noise , Gaussian noise Used for the t-1th step of noise addition, and then use the stable diffusion model to add noise formula x t-1 , and get the noise added The noisy target image x t-1 ; Among them, a set of parameters are predefined in the stable diffusion model , Refers to a set of parameters The tth value in refers to ; Step 2.1.2, obtain random Gaussian noise For the current noisy target image x that has been noisy for t-1 steps t-1 Perform another single-step noise addition, the noise addition formula is: ， Get the image after adding noise , while preserving the Gaussian noise As real noise; Refers to a set of predefined parameters in the stable diffusion model The tth data in ; Step 2.1.3: Use the CLIP model to perform word embedding on the text. First, split the text into independent words or subword units, and then embed the units through the text encoder of the CLIP model to generate a text vector T corresponding to the text semantics. text ; Step 2.1.4, transform the text vector T text , the image after adding noise and the current time step t are passed into the stable diffusion model combined with LoRA, allowing the model to predict the amount of noise added at step t , for the image Remove Step noise, use the following formula to get the noisy image: 。 5. The method according to claim 4, characterized in that Step 2.2 includes: Step 2.2.1, after obtaining the noisy image After that, the noisy image is first converted to The decoder of the VAE model restores the size to obtain the original size image with noise ; Step 2.2.2: Use CLIP’s image encoder to decode the noisy original size image. Extract feature vectors and convert the original noisy image into Convert to image vector I.

6. The method according to claim 5, characterized in that Step 2.3 includes: Step 2.3.1, use the loss formula For the text vector T text And the image vector I is used for loss calculation, where Replaced by T, By replacing I, we get As the semantic similarity between the text vector and the image vector I; Step 2.3.2, determine the weight setting at this time according to the current time step t , through the formula , adjust the weight value; In step 2.3.3, the CLIP loss is introduced into the gradient calculation graph, and the calculated CLIP loss gradient is passed back along the optimization path to adjust the parameters in LoRA.

7. The method according to claim 6, characterized in that Step 3 includes: Step 3.1, use a perceptual loss function to optimize the network results, the formula is: ， ， in represents the total loss value of the perceived loss, Indicates the feature extraction of the image by each layer of convolutional network. represents the dynamic weight term; exp is the natural exponential function; represents the feature extraction network of the kth layer; Step 3.2, for noisy images After resizing, use perceptual loss for calculation; Step 3.3, using perceptual loss For the original noisy image The perceptual loss is calculated by calculating the original size image with noise The difference between the target fine-tuning image X0 and the layers in the perception model is used to guide the adjustment of LoRA internal parameters during back propagation.

8. The method according to claim 7, characterized in that Step 3.3 specifically includes: Step 3.3.1, the feature extraction network VGG16 is layered, and the outputs of the 0th, 1st, 2nd, 3rd, 4th, and 5th layers of convolution are selected as the five feature vectors of the image; a linear layer is added to the output features, and a linear change is performed to kill neurons with a set probability; the input channels of the feature extraction network VGG16 are adjusted to 64, 128, 256, 512, 512; Step 3.3.2: Convert the original size noisy target image The target fine-tuned image X0 is input into the feature extraction network VGG16. During the calculation process, the feature extraction network VGG16 saves the calculation results in the 0th, 1st, 2nd, 3rd, 4th, and 5th layers for calculating the loss. Step 3.3.3, calculate the original size noisy target image And the second-order norm loss of the target fine-tuning image X0 after passing through different feature extraction layers , and record the loss value, the formula is: , Calculate the weighted value of each layer and save it, and finally use the formula to sum the total loss: ， in represents the perceptual loss term.

9. The method according to claim 8, characterized in that Step 4 includes: defining the following final loss function: ， in, refers to the total loss value, is the CLIP loss, For perceptual loss, when the CLIP loss enabling condition is not met, the parameter Set to 0; after CLIP loss is enabled, Set to 1.

10. A LoRA fine-tuning optimization system for a stable diffusion model based on CLIP loss and perception loss implemented according to the method described in any one of claims 1 to 9, characterized in that: include: The CLIP loss calculation module is used to: encode the text corresponding to the target fine-tuning image to obtain the text vector, and then After adding noise for t steps, the stable diffusion model combined with LoRA is used to predict the noise of the tth step, and then the predicted noise is subtracted to obtain the noisy image ; Then the noisy image Use the decoder of the VAE model to restore the size, then use the CLIP model to encode the restored image into a vector, and finally use the adaptive weight loss calculation formula to calculate the difference between the text vector and the image vector; The perceptual loss calculation module is used to: firstly calculate the target fine-tuned image X0 and the noisy image from which the noise of the tth step has been removed Input the feature extraction network VGG16, specify to extract features on the 0th, 1st, 2nd, 3rd, 4th, and 5th convolution layers in VGG16, and use the adaptive perceptual loss to calculate the features extracted above; finally, after merging the CLIP loss and the perceptual loss with the basic loss, back propagate to guide the parameter update inside LoRA.

Images