Efficient parameter fine tuning method for specific scene text generation image model

By freezing and decomposing the weights of the pretrained model, combining LoRA and feature scaling factors, the problems of low efficiency and overfitting in fine-tuning of text generation image models are solved, and text generation with high stability, flexibility and strong generalization capabilities are achieved Image model.

CN118939825APending Publication Date: 2024-11-12ZHENGZHOU UNIV

Patent Information

Application Number
CN202410908869.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-08
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The existing fine-tuning methods of text generation image models have problems such as inefficiency, waste of resources and overfitting, especially in complex task scenarios, which are difficult to improve the stability and generalization capabilities of the model.

Method used

By freezing the weights of the pre-trained model, decompose them into amplitude vectors and direction matrices, and adding regular terms during multi-dimensional adjustment, combining the Hadamama product of LoRA low-rank matrix to update the weights, and introducing feature scaling factors to balance the Unet network and jump connections The feature contribution of the model ensures that the model can generate images efficiently in specific scenarios.

Benefits of technology

It improves the stability, flexibility and reasoning efficiency of the model, alleviates the impact of language drift and overfitting, improves the diversity and fidelity of the image, and enhances the generalization ability of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939825A_ABST
    Figure CN118939825A_ABST
Patent Text Reader

Abstract

The invention discloses an efficient parameter fine tuning method for a specific scene text generation image model, and the method comprises the following steps: S1, carrying out weight freezing: carrying out the weight freezing of a pre-training model, and obtaining a frozen weight; s2, weight decomposition: decomposing the frozen weight, and decoupling the frozen weight into an amplitude vector and a direction matrix; s3, model weight updating: performing multi-dimensional adjustment on the weight, and adding regular terms in the adjustment process to guide parameter updating to obtain an updated weight; s4, weight merging: merging the frozen weight in the step S1 and the updated weight in the step S3; and S5, introducing a feature scaling factor: in the generation task application process, improving the quality of the generated image through the action of the modulation factor in the Unet network of the pre-training model. According to the method, the stability of model training, the flexibility of parameter adjustment and the reasoning efficiency can be improved, the influence of language drift and overfitting is relieved, and the method has high-fidelity image characteristics and strong generalization ability.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and machine learning technology, and in particular to a method for efficiently fine-tuning parameters of a text-generated image model for a specific scene. Background Art

[0002] Text-generated images refer to the generation of images based on text prompts provided by users. Text-generated image models have demonstrated their wide application value in many fields, such as creative arts, game design, e-commerce advertising, home design, and education industries. Training these models for different scenarios usually requires a large amount of data, computing power, and time, which is very expensive.

[0003] The metrics used to measure the performance of text-to-image models are fidelity, controllability, diversity, and image quality.

[0004] Efficient parameter fine-tuning is a low-cost text-to-image model training method. Existing fine-tuning methods include: Adapter-based method, Prompt method, and Low-Rank Adaptation method.

[0005] The adapter-based method inserts an adapter (neural network) after the multi-head attention and feedforward layers in the Transformer architecture. The parameters of the pre-trained model are frozen during fine-tuning, and only the parameters in the adapter are trained. Although the adapter method greatly reduces the number of training parameters, it introduces additional complexity by changing the network structure of the model. For some application scenarios with highly complex tasks, additional delays will be introduced when the model generates images, resulting in low efficiency.

[0006] The prompt-based fine-tuning method adapts the model to different tasks by changing the embedding vector of the prompt word. Under the guidance of language, new concepts specified by the user are generated and encoded into the intermediate representation of the pre-trained text-to-image model. However, since the embedding vector of the prompt word of each task must be fine-tuned, it will cause inefficiency and waste of resources.

[0007] The LoRA-based fine-tuning method adds a bypass next to the frozen pre-trained weights. The bypass consists of two low-rank matrices. During fine-tuning, only the low-rank matrix is ​​updated. When adapting to a specific task, the weight of the corresponding task is switched based on the pre-trained weights. Since the structure of the model is not changed, the inference delay is not introduced, and the efficiency of generating images is greatly improved. This method is also based on the idea of ​​LoRA. Summary of the invention

[0008] The technical problem to be solved by the present invention is to provide a parameter efficient fine-tuning method for a text generation image model for specific scenarios, which can improve the stability of model training, the flexibility of parameter adjustment and the reasoning efficiency, alleviate the impact of language drift and overfitting, and have high-fidelity image characteristics and strong generalization capabilities.

[0009] In order to achieve the above object, the technical solution adopted by the present invention is: a method for efficiently fine-tuning parameters of a text-generated image model for a specific scene, characterized in that it includes the following steps:

[0010] S1. Weight freezing: The weights of the pre-trained model are frozen to obtain frozen weights;

[0011] S2. Weight decomposition: decompose the frozen weights into magnitude vectors and direction matrices;

[0012] S3. Model weight update: adjust the weights in multiple dimensions, and add regularization terms to guide parameter updates during the adjustment process to obtain updated weights;

[0013] S4. Weight merging: merging the frozen weights in step S1 with the updated weights in step S3;

[0014] S5. Introducing feature scaling factors: In the application process of the generation task, the quality of the generated image is improved by modulating the factor in the Unet network of the pre-trained model.

[0015] As a preferred technical solution of the present invention, in step S2, the frozen weight matrix The decomposition of is adopted as formula (1)

[0016]

[0017] in, is the frozen weight matrix, is the magnitude vector, is the direction matrix, ‖·‖ c is the vector norm of each column of the direction matrix.

[0018] As a preferred technical solution of the present invention, in step S2, the frozen weight matrix Corresponding to the query matrix and value matrix in the self-attention module in Unet, after weight decomposition, the magnitude vector m and direction matrix V are updated respectively through step S3;

[0019] In step S3, the decomposed direction matrix V and amplitude vector m are first initialized, and then the direction matrix is ​​updated by the Hadamard product form of two sets of LoRA low-rank matrices, and the amplitude vector m is updated by the gradient descent algorithm. The updated weight matrix can be expressed as formula (2),

[0020]

[0021] Among them, W' initializes the direction matrix V to the pre-trained weight matrix W0, the amplitude vector m is initialized to the column vector norm of the pre-trained weight matrix W0, ΔV is the update increment of the direction matrix, which is obtained by the Hadamard product of two sets of low-rank matrices, that is, ΔV = B1A1⊙B2A2, where They are all trainable parameter matrices. The initialization values ​​of A1 and A2 follow a Gaussian distribution with a mean of zero, and the initialization values ​​of B1 and B2 follow a zero distribution. V+ΔV is the direction matrix of the updated weight matrix, ||V+ΔV|| c is the norm of each column of the updated direction matrix, ensuring that each column of the updated direction matrix is ​​a unit vector, and each scalar in the direction vector m is the amplitude of each column vector in the V+ΔV matrix. After the direction matrix V is updated, the updated direction matrix V+ΔV is kept unchanged, and the gradient of the loss function relative to the amplitude vector m is calculated by the back propagation algorithm, and the parameter m is updated by the gradient descent algorithm;

[0022] In order to reduce memory consumption during training, ||V+ΔV||| c Treated as a constant, it is separated from the gradient graph during backpropagation, which means that ||V+ΔV|| c Although it dynamically reflects the update of ΔV, it does not receive any gradients in the back propagation. Therefore, according to formula (2), the loss function can be obtained: The gradients of V'=V+ΔV and m are shown in equations (3) and (4):

[0023]

[0024]

[0025] Among them, V'=V+ΔV represents the updated direction matrix, is the loss function With respect to the gradient of the direction matrix V'=V+ΔV, is the loss function The gradient of the weight matrix W' is: is the loss function The gradient of the updated magnitude vector m.

[0026] As a preferred technical solution of the present invention, the addition of a regularization term in the loss function during the adjustment process of step S3 is to use the prior knowledge of the pre-trained model to generate pictures of the same category as the main body of the training set, and use these pictures to supervise the fine-tuning model, so as to retain the prior knowledge of the category at the beginning of fine-tuning, and ensure that the model will not be overly focused on a small number of training samples and ignore the broad characteristics of the category during fine-tuning;

[0027] The loss function of the pre-trained model is shown in formula (5):

[0028]

[0029] where x is the ground truth of the subject image, c is the text condition containing the subject’s unique identifier, ∈ is the noise sampled from a standard normal distribution, t is the time step, and z t is the potential representation of the subject noise image, ∈ θ (·) is the network where the model predicts the noise at each time step. The loss function uses the mean square error (MSE) between the noise predicted by the model and the true noise to train the model. The prior preservation loss for a specific category is defined as shown in formula (6):

[0030]

[0031] Here z' t is the potential representation of the category noise image, c' is the condition containing the category noun, ∈ θ (·) is the noise predicted by the fine-tuned model at each time step;

[0032] Both loss functions use the mean square error between the noise predicted by the model and the real noise to train the model. The difference is that the prior loss function for a specific category uses the category image and category prompt words of the training sample body to supervise the fine-tuning model. The combined loss function is shown in formula (7):

[0033] L=L SD +L PPL (7).

[0034] As a preferred technical solution of the present invention, in step S5, the task application is generated by the merged model, and the backbone feature scaling factor b and the jump feature scaling factor s are introduced in the inference stage to re-weight the feature contributions of the Unet backbone network and the jump connection respectively, so as to balance the feature contributions of the Unet backbone network and the jump connection, thereby improving the image quality.

[0035] As a preferred technical solution of the present invention, the process of introducing the backbone feature scaling factor b is to use an adaptive factor according to the specific characteristics of the sample features, and the specific steps of introducing the backbone feature scaling factor b include:

[0036] K1. The backbone feature map x of the decoder layer l l Take the average over all channels, as shown in formula (8);

[0037]

[0038] Among them, x l,i is the backbone feature map x of the decoder layer l l The feature map of the ith channel, C is the number of channels of the feature map; K2. Backbone feature factor map α l The determination method is shown in formula (9):

[0039]

[0040] Among them, α l represents the backbone feature factor graph, b l is the backbone feature scaling factor;

[0041] K3. In order to avoid the backbone feature factor graph α l Indiscriminately acting on all channels will cause excessive smoothing of the image texture. Therefore, the backbone feature factor map α l Acting on half of the channel, as shown in formula (10)

[0042]

[0043] As a preferred technical solution of the present invention, the jump feature factor s is introduced by applying a fast Fourier transform to convert the jump connection feature map into the frequency domain, the jump feature factor s then scales these frequency domain features, and finally converts the jump connection back to the spatial domain through an inverse Fourier transform.

[0044] As a preferred technical solution of the present invention, the working process of the jump feature scaling factor s is shown in formulas (11)-(14):

[0045]

[0046]

[0047]

[0048]

[0049] Among them, FFT(·) is Fourier transform, IFFT(·) is inverse Fourier transform, ⊙ is element-by-element multiplication, β l,i is the Fourier mask, which is a function that takes different values ​​according to different frequency values ​​r. thresh is the threshold frequency, s rFor the scaling factors of different frequency components, the low-frequency components are weakened and the high-frequency components remain unchanged.

[0050] The beneficial effects of adopting the above technical solution are: 1) High stability and flexibility: By decoupling the pre-trained weights into two components, amplitude and direction, the weights are adjusted in multiple dimensions, which improves the stability of model training and the flexibility of parameter adjustment.

[0051] 2) High inference efficiency: This method does not change the original model structure, but only makes small adjustments to some parameters, so it does not add extra computational burden. This means that the model runs very quickly and does not slow down processing.

[0052] 3) Good portability: Compared with the several GB checkpoints of fully fine-tuned, the weight checkpoints obtained by the PEFT method are only tens of MB. This makes it possible to switch the corresponding weight checkpoints according to different tasks without replacing the entire model.

[0053] 4) High diversity: By introducing regularization terms in the loss function, the model’s over-dependence on training samples is reduced, the impact of language drift and overfitting is alleviated, and the diversity of generated images is improved.

[0054] 5) High fidelity: By introducing two modulation factors in the inference stage to balance the contribution of different features in the UNET network, the quality of the image is improved.

[0055] 6) Strong generalization ability: The Hadamard product of the matrix increases the range of the rank of the LoRA learnable parameter space, which enhances the model’s ability to learn knowledge and thus improves its performance in new and unknown situations. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0057] Figure 1 It is a general implementation flow chart of the present invention;

[0058] Figure 2 It is a weight decomposition method diagram of the present invention;

[0059] Figure 3 It is a diagram showing the modulation process of the backbone feature scaling factor b and the jump connection feature scaling factor s on the feature graph of the present invention. DETAILED DESCRIPTION

[0060] See attached Figure 1-3 , a specific embodiment of the present invention comprises the following steps:

[0061] S1. Weight freezing: The weights of the pre-trained model are frozen to obtain frozen weights;

[0062] S2. Weight decomposition: decompose the frozen weights and decouple them into magnitude vectors and direction matrices; prepare for multi-dimensional adjustment of weights;

[0063] S3. Model weight update: The weights are adjusted in multiple dimensions, and regularization terms are added during the adjustment process to guide parameter updates to obtain updated weights. The main task of adding regularization terms is to add the regularization term of category prior loss to the loss function of the fine-tuning method to prevent the fine-tuning model from overfitting to the training set images, thereby improving the generalization ability of the model. Multi-dimensional adjustment after weight decomposition can improve the flexibility and stability of parameter updates.

[0064] S4. Weight merging: merging the frozen weights in step S1 with the updated weights in step S3;

[0065] S5. Introducing feature scaling factors: In the process of applying the generation task, the quality of the generated image is improved in the Unet network of the pre-trained model through feature scaling. The main task of introducing feature scaling factors is to add two feature scaling factors to the Unet structure of the pre-trained model to improve the quality of the model-generated image.

[0066] See attached separately Figure 2 Steps S2 and S3 are shown, in which the weight matrix is ​​frozen in step S2. The decomposition of is adopted as formula (1),

[0067]

[0068] in, is the frozen weight matrix, is the magnitude vector, is the direction matrix, ‖·‖ c is the vector norm of each column of the direction matrix.

[0069] In step S2, the frozen weight matrix Corresponding to the query matrix and value matrix in the self-attention module in Unet, after weight decomposition, the magnitude vector m and direction matrix V are updated respectively through step S3;

[0070] In step S3, the decomposed direction matrix V and amplitude vector m are first initialized, and then the direction matrix is ​​updated through the Hadamard product form of two sets of LoRA low-rank matrices to improve the rank of the low-rank space and the generalization ability of the model. Compared with the rank r of the original set of LoRA low-rank matrices, the rank of this method can reach r 2, which significantly increases the upper limit of the rank, so that more knowledge and capabilities can be learned and the model performance can be improved; and the amplitude vector m is updated by the gradient descent algorithm. The updated weight matrix can be expressed as formula (2),

[0071]

[0072] Among them, W' initializes the direction matrix V to the pre-trained weight matrix W0, the amplitude vector m is initialized to the column vector norm of the pre-trained weight matrix W0, ΔV is the update increment of the direction matrix, which is obtained by the Hadamard product of two sets of low-rank matrices, that is, ΔV = B1A1⊙B2A2, where They are all trainable parameter matrices. The initialization values ​​of A1 and A2 follow a Gaussian distribution with a mean of zero, and the initialization values ​​of B1 and B2 follow a zero distribution. V+ΔV is the direction matrix of the updated weight matrix, ||V+ΔV|| c is the norm of each column of the updated direction matrix, ensuring that each column of the updated direction matrix is ​​a unit vector, and each scalar in the direction vector m is the amplitude of each column vector in the V+ΔV matrix. After the direction matrix V is updated, the updated direction matrix V+ΔV is kept unchanged, and the gradient of the loss function relative to the amplitude vector m is calculated by the back propagation algorithm, and the parameter m is updated by the gradient descent algorithm;

[0073] Since DoRA redirects low-order adaptation to the directional component, the gradient ΔW of the low-order update is different from the gradient of W', and additional memory is required to handle this difference in backpropagation. To reduce memory consumption during training, ||V+ΔV|| is replaced by ||V+ΔV|| in the gradient calculation. c Treated as a constant, it is separated from the gradient graph during backpropagation, which means that ||V+ΔV|| c Although it dynamically reflects the update of ΔV, it does not receive any gradients in the back propagation. Therefore, according to formula (2), the loss function can be obtained: The gradients of V'=V+ΔV and m are shown in equations (3) and (4):

[0074]

[0075]

[0076] Among them, V'=V+ΔV represents the updated direction matrix, is the loss function With respect to the gradient of the direction matrix V'=V+ΔV, is the loss function The gradient of the weight matrix W' is: is the loss function The gradient of the updated magnitude vector m.

[0077] Introducing a category-specific prior preservation loss (PPL) into the loss function prevents the model from overfitting and associating the subject of this category with a specific instance. When fine-tuning the text encoder, the problem of language drift will arise, resulting in the fine-tuning model being unable to generate subjects of the same type as the training set and a reduction in the diversity of generated subjects. Adding a regularization term to the loss function during the adjustment process of step S3 is to use the prior knowledge of the pre-trained model to generate pictures of the same category as the subject of the training set, and use these pictures to supervise the fine-tuning model so as to retain the prior knowledge of the category at the beginning of fine-tuning, ensuring that the model will not be overly focused on a small number of training samples and ignore the broad features of the category during fine-tuning;

[0078] The loss function of the pre-trained model is shown in formula (5):

[0079]

[0080] where x is the ground truth of the subject image, c is the text condition containing the subject’s unique identifier, ∈ is the noise sampled from a standard normal distribution, t is the time step, and z t is the potential representation of the subject noise image, ∈ θ (·) is the network where the model predicts the noise at each time step. The loss function uses the mean square error (MSE) between the noise predicted by the model and the true noise to train the model. The prior preservation loss for a specific category is defined as shown in formula (6):

[0081]

[0082] Here z' t is the potential representation of the category noise image, c' is the condition containing the category noun, ∈ θ (·) is the noise predicted by the fine-tuned model at each time step;

[0083] Both loss functions use the mean square error between the noise predicted by the model and the real noise to train the model. The difference is that the prior loss function for a specific category uses the category image and category prompt words of the training sample body to supervise the fine-tuning model. The combined loss function is shown in formula (7):

[0084] L=L SD +L PPL (7)

[0085] Introducing the feature scaling factor part: Unet is a key component in the Stable Diffusion model, which is used to gradually remove noise from the image. The backbone network of UNet is responsible for the basic denoising process, while the skip connection provides high-frequency features from different encoding layers to the decoding layer. These high-frequency features propagate fine-grained semantic information and detail features. This information may be lost in the process of layer-by-layer transmission. The edge and semantic information of the image can be retained through the skip connection, which has a great effect on the generation quality and clarity of the image. However, the unexpected consequence of the skip connection is that the inherent denoising ability of the backbone network will be weakened in the inference stage. In order to more effectively utilize the advantages of these two parts, two feature scaling factors are introduced in the inference stage to re-weight the feature contributions of the two parts to balance the feature contributions of the Unet backbone and the skip connection, thereby improving the image quality.

[0086] The backbone feature factor b is designed to amplify the backbone feature map and enhance the denoising ability. Adding the backbone feature factor can significantly enhance the denoising ability, but it will over-smooth the image texture, while the jump feature factor s can alleviate the problem of over-smoothing the image texture by the backbone network. By applying the fast Fourier transform (FFT) to transform the jump connection feature map to the frequency domain, the jump feature factor s scales these frequency domain features, and then converts them back to the spatial domain through the inverse Fourier transform (IFFT). Such a processing strategy can more effectively control and adjust the details and texture information of the image by modifying the amplitude of different frequency components.

[0087] See attached separately Figure 3 In step S5, the task application is generated by the merged model, and the backbone feature scaling factor b and the jump feature scaling factor s are introduced in the inference stage to re-weight the feature contributions of the Unet backbone network and the jump connection respectively, so as to balance the feature contributions of the Unet backbone network and the jump connection, thereby improving the image quality.

[0088] The process of introducing the backbone feature scaling factor b is to use an adaptive factor according to the specific characteristics of the sample features. The specific steps of introducing the backbone feature scaling factor b include:

[0089] K1. The backbone feature map x of the decoder layer l l Take the average over all channels, as shown in formula (8);

[0090]

[0091] Among them, x l,i is the backbone feature map x of the decoder layer l l The feature map of the i-th channel, C is the number of channels of the feature map;

[0092] K2. Backbone feature factor graph α lThe determination method is shown in formula (9):

[0093]

[0094] Among them, α l represents the backbone feature factor graph, b l is the backbone feature scaling factor;

[0095] K3. In order to avoid the backbone feature factor graph α l Applying this to all channels indiscriminately will result in over-smoothing of the image texture.

[0096] Therefore, the backbone feature factor graph α l Acting on half of the channel, as shown in formula (10)

[0097]

[0098] In order to further effectively alleviate the problem of over-smoothing of image texture, the jump connection feature scaling factor s is introduced to selectively weaken its low-frequency components in the Fourier frequency domain. The high-frequency components of the image correspond to the detailed information such as the texture and edge of the image, and the low-frequency components correspond to the global vision and overall layout of the image. In Unet denoising, the low-frequency components do not change much during the image generation process, while the high-frequency components change significantly. Therefore, the jump feature factor can reduce the impact of over-smoothing of image texture to a certain extent by suppressing the low-frequency components to varying degrees in the Fourier domain and keeping the high-frequency components unchanged. The jump feature scaling factor s is introduced to convert the jump connection feature map to the frequency domain by applying the fast Fourier transform, and the jump feature scaling factor s then scales these frequency domain features, and finally converts the jump connection feature map back to the spatial domain through the inverse Fourier transform.

[0099] The working process of the jump feature scaling factor s is shown in formulas (11)-(14)

[0100]

[0101]

[0102]

[0103]

[0104] Among them, FFT(·) is Fourier transform, IFFT(·) is inverse Fourier transform, ⊙ is element-by-element multiplication, β l,i is the Fourier mask, which is a function that takes different values ​​according to different frequency values ​​r. thresh is the threshold frequency, s rThe scaling factors for different frequency components are used to weaken the low-frequency components and keep the high-frequency components unchanged. Both feature scaling factors are adaptively set during the inference phase, without the need for training or inference, and only a few lines of code are needed to achieve high-quality image generation.

[0105] The above description is only proposed as an implementable technical solution of the present invention, and is not intended to be a single limitation on the technical solution itself.

Claims

1. An efficient parameter fine-tuning method for scene-specific text generation image model, characterized in that: The following steps are involved: S1. Weight freezing: The weights of the pre-trained model are frozen to obtain frozen weights; S2. Weight decomposition: decompose the frozen weights into magnitude vectors and direction matrices; S3. Model weight update: adjust the weights in multiple dimensions, and add regularization terms to guide parameter updates during the adjustment process to obtain updated weights; S4. Weight merging: merging the frozen weights in step S1 with the updated weights in step S3; S5. Introducing feature scaling factors: In the application process of the generation task, the quality of the generated image is improved by modulating the factor in the Unet network of the pre-trained model.

2. The efficient parameter fine-tuning method according to claim 1, characterized in that: In step S2, the frozen weight matrix The decomposition of is adopted as formula (1), in, is the frozen weight matrix, is the magnitude vector, is the direction matrix, ‖·‖ c is the vector norm of each column of the direction matrix.

3. The efficient parameter fine-tuning method according to claim 2, characterized in that: In step S2, the frozen weight matrix Corresponding to the query matrix and value matrix in the self-attention module in Unet, after weight decomposition, the magnitude vector m and direction matrix V are updated respectively through step S3; In step S3, the decomposed direction matrix V and amplitude vector m are first initialized, and then the direction matrix is ​​updated by the Hadamard product form of two sets of LoRA low-rank matrices, and the amplitude vector m is updated by the gradient descent algorithm. The updated weight matrix can be expressed as formula (2), Among them, W' initializes the direction matrix V to the pre-trained weight matrix W0, the amplitude vector m is initialized to the column vector norm of the pre-trained weight matrix W0, ΔV is the update increment of the direction matrix, which is obtained by the Hadamard product of two sets of low-rank matrices, that is, ΔV = B1A1⊙B2A2, where They are all trainable parameter matrices. The initialization values ​​of A1 and A2 follow a Gaussian distribution with a mean of zero, and the initialization values ​​of B1 and B2 follow a zero distribution. V+ΔV is the direction matrix of the updated weight matrix, ||V+ΔV|| c is the norm of each column of the updated direction matrix, ensuring that each column of the updated direction matrix is ​​a unit vector, and each scalar in the direction vector m is the amplitude of each column vector in the V+ΔV matrix. After the direction matrix V is updated, the updated direction matrix V+ΔV is kept unchanged, and the gradient of the loss function relative to the amplitude vector m is calculated by the back propagation algorithm, and the parameter m is updated by the gradient descent algorithm; In order to reduce memory consumption during training, ||V+ΔV||| c Treated as a constant, it is separated from the gradient graph during backpropagation, which means that ||V+ΔV|| c Although it dynamically reflects the update of ΔV, it does not receive any gradients in the back propagation. Therefore, according to formula (2), the loss function can be obtained: The gradients of V'=V+ΔV and m are shown in equations (3) and (4): Among them, V'=V+ΔV represents the updated direction matrix, is the loss function With respect to the gradient of the direction matrix V'=V+ΔV, is the loss function The gradient of the weight matrix W' is: is the loss function The gradient of the updated magnitude vector m.

4. The efficient parameter fine-tuning method according to claim 1, characterized in that: Adding a regularization term to the loss function during the adjustment process of step S3 is to use the prior knowledge of the pre-trained model to generate pictures of the same category as the main body of the training set, and use these pictures to supervise the fine-tuning model, so as to retain the prior knowledge of the category at the beginning of fine-tuning, and ensure that the model will not be overly focused on a small number of training samples and ignore the broad characteristics of the category during fine-tuning; The loss function of the pre-trained model is shown in formula (5): where x is the ground truth of the subject image, c is the text condition containing the subject’s unique identifier, ∈ is the noise sampled from a standard normal distribution, t is the time step, and z t is the potential representation of the subject noise image, ∈ θ (·) is the network where the model predicts the noise at each time step. The loss function uses the mean square error (MSE) between the noise predicted by the model and the true noise to train the model. The prior preservation loss for a specific category is defined as shown in formula (6): Here z' t is the potential representation of the category noise image, c' is the condition containing the category noun, ∈ θ (·) is the noise predicted by the fine-tuned model at each time step; Both loss functions use the mean square error between the model-predicted noise and the real noise to train the model. The difference is that the category-specific prior loss function uses the category images and category prompt words of the training sample body to supervise the fine-tuning model. The combined loss function is shown in formula (7): L=L SD +L PPL (7) 5. The efficient parameter fine-tuning method according to claim 1, characterized in that: In step S5, the task application is generated by the merged model, and the backbone feature scaling factor b and the jump feature scaling factor s are introduced in the inference stage to re-weight the feature contributions of the Unet backbone network and the jump connection respectively, so as to balance the feature contributions of the Unet backbone network and the jump connection, thereby improving the image quality.

6. The efficient parameter fine-tuning method according to claim 5, characterized in that: The process of introducing the backbone feature scaling factor b is to use an adaptive factor according to the specific characteristics of the sample features. The specific steps of introducing the backbone feature scaling factor b include: K1. The backbone feature map x of the decoder layer l l Take the average over all channels, as shown in formula (8); Among them, x l,i is the backbone feature map x of the decoder layer l l The feature map of the i-th channel, C is the number of channels of the feature map; K2. Backbone feature factor graph α l The determination method is shown in formula (9): Among them, α l represents the backbone feature factor graph, b l is the backbone feature scaling factor; K3. In order to avoid the backbone feature factor graph α l Indiscriminately acting on all channels will cause excessive smoothing of the image texture. Therefore, the backbone feature factor map α l Acting on half of the channel, as shown in formula (10) 7. The efficient parameter fine-tuning method according to claim 5, characterized in that: The jump feature scaling factor s is introduced to convert the jump connection feature map to the frequency domain by applying the fast Fourier transform, and the jump feature scaling factor s then scales these frequency domain features, and finally converts the jump connection feature map back to the spatial domain through the inverse Fourier transform.

8. The efficient parameter fine-tuning method according to claim 7, characterized in that: The working process of the jump feature scaling factor s is shown in formulas (11)-(14) Among them, FFT(·) is Fourier transform, IFFT(·) is inverse Fourier transform, ⊙ is element-by-element multiplication, β l,i is the Fourier mask, which is a function that takes different values ​​according to different frequency values ​​r. thresh is the threshold frequency, s r For the scaling factors of different frequency components, the low-frequency components are weakened and the high-frequency components remain unchanged.

Citation Information

Patent Citations

  • General image text description generation method based on multi-task learning and related device

    CN116912629A

  • Image generation method, device and equipment and computer readable storage medium

    CN117475038A

  • Texture-coordinated MRI (Magnetic Resonance Imaging) image reconstruction method based on TC-DiffReccon model

    CN117689750A

  • Image generation method and device, equipment and storage medium

    CN117934646A

  • Text illustration generation method and related device

    CN118052907A

Cited By

  • Incremental model merging method and system for large language model

    CN119294465A