A diffusion model fine-tuning method, device, equipment and readable storage medium

Abstract

A diffusion model fine-tuning method, device, equipment and readable storage medium, comprising: obtaining a training data set of a target subject, encoding the training sample to obtain a real latent space feature, and constructing a noisy latent space feature according to the real latent space feature; load the pre-trained diffusion model, input the noisy latent space feature into the pre-trained diffusion model for forward propagation to obtain the predicted noise; determine the composite loss according to the training sample, the noisy latent space feature and the predicted noise, including the basic reconstruction loss, the structure loss based on the normalized space attention weighted and the edge consistency loss based on the channel level grouping edge detection; update the parameters of the pre-trained diffusion model based on the composite loss until the model converges to obtain the fine-tuned diffusion model. The application guarantees the structural consistency of the generated result through the structure loss, and enhances the edge definition and detail texture of the generated image through the edge consistency loss, so as to obtain a high-fidelity diffusion model.

Description

Technical Field

[0001] This application relates to the field of image generation, specifically to a diffusion model fine-tuning method, apparatus, device, and readable storage medium. Background Technology

[0002] As a mainstream image generation technology, diffusion models are widely used in content creation and personalized customization scenarios in vertical fields. With the increasing depth of application, users not only require the generated images to conform to the semantic content of the prompt words, but also put forward extremely high technical requirements for the structural integrity, edge sharpness, and detail reproduction of specific subjects (such as industrial parts, specific vehicles, or people). There is an urgent need for a fine-tuning scheme that can retain the general generation capabilities of the model while faithfully reproducing the features of the target subject.

[0003] Existing fine-tuning methods for diffusion models typically employ a latent space-based denoising training paradigm. This involves encoding training samples as real latent space features and adding noise to construct noisy features. These noisy features are then input into a pre-trained model for forward propagation to obtain predicted noise. The model parameters are updated by minimizing the mean squared error between the predicted and real noise. While some existing solutions introduce additional perceptual or semantic alignment losses, they primarily rely on global error calculations to drive model optimization.

[0004] However, existing technologies still have shortcomings when fine-tuning vertical domains with high structural accuracy requirements. First, existing loss functions typically prioritize overall reconstruction error during optimization, making it difficult to guarantee the feature learning priority of key structural regions. This leads to deformation of key structural features in the generated results when dealing with complex geometric structures. Second, existing technologies lack effective mechanisms for capturing and constraining edge contour information in the feature space, resulting in blurring of high-frequency details and edge contours in the generated images. Finally, existing optimization methods struggle to simultaneously address the requirements of reconstruction accuracy and structural detail, affecting the final quality of the generated model and requiring improvement. Summary of the Invention

[0005] This application provides a diffusion model fine-tuning method, apparatus, device, and readable storage medium, which can solve the technical problems of easy distortion of the main structure and blurred edge details in the vertical domain fine-tuning process of the diffusion model in the prior art.

[0006] In a first aspect, embodiments of this application provide a diffusion model fine-tuning method, including: Obtain the training dataset of the target subject, encode the training samples in the training dataset to obtain the true latent space features, and construct noisy latent space features based on the true latent space features; Load the pre-trained diffusion model, input the noisy latent space features into the pre-trained diffusion model for forward propagation, and obtain the predicted noise; The composite loss is determined based on training samples, noisy latent space features, and predicted noise. The composite loss includes basic reconstruction loss, structural loss based on normalized spatial attention weighting, and edge consistency loss based on channel-level grouped edge detection. The parameters of the pre-trained diffusion model are updated based on the composite loss until the model converges, resulting in a fine-tuned diffusion model.

[0007] In conjunction with the first aspect, in one implementation, the structural loss based on normalized spatial attention weighting is obtained as follows: Construct a spatial attention weight map of training samples in the latent space; The spatial attention weight map is normalized, and then the normalized spatial attention weight map is multiplied element-wise with the feature-by-feature tensor corresponding to the basic reconstruction loss to obtain the weighted structural loss.

[0008] In conjunction with the first aspect, in one implementation, constructing a spatial attention weight map of training samples in the latent space includes: Construct a Gaussian distribution mask for the target subject in the training samples within the latent space; Weights are assigned based on a Gaussian distribution mask, making the weight of the target area higher than that of the background area, thus obtaining a spatial attention weight map.

[0009] In conjunction with the first aspect, in one implementation, the edge consistency loss based on channel-level grouped edge detection is obtained as follows: The characteristics of the predicted denoising latent space are determined based on the characteristics of the noisy latent space and the predicted noise. Edge detection is performed on the predicted denoised latent space features and the real latent space features in each channel to obtain the gradient magnitude of the predicted channel and the gradient magnitude of the real channel. The channel-level grouped gradient difference between the predicted channel gradient magnitude and the actual channel gradient magnitude is calculated, and the channel-level grouped gradient difference is globally averaged to obtain the edge consistency loss.

[0010] In conjunction with the first aspect, in one implementation, edge detection is performed on the predicted denoised latent space features and the true latent space features in each channel to obtain the gradient magnitude of the predicted channel and the gradient magnitude of the true channel, including: For the predicted denoised latent space features and the real latent space features, depthwise separable convolutions are performed on the X and Y axes based on the Sobel operator in each channel to obtain the X-axis gradient magnitude component and Y-axis gradient magnitude component of each channel. The X-axis gradient magnitude component and Y-axis gradient magnitude component of each corresponding channel are combined to obtain the predicted channel gradient magnitude and the true channel gradient magnitude of each channel.

[0011] In conjunction with the first aspect, in one implementation, determining the composite loss based on training samples, noisy latent space features, and prediction noise includes: In each training step, the predicted noise and the predicted denoised latent space features are reused to compute the basic reconstruction loss, the structure loss based on normalized spatial attention weighting, and the edge consistency loss based on channel-level grouped edge detection in parallel. The predicted denoised latent space features are determined based on the noisy latent space features and the predicted noise. The composite loss is obtained by combining the basic reconstruction loss, structural loss, and edge consistency loss.

[0012] In conjunction with the first aspect, in one implementation method, the diffusion model fine-tuning method further includes: Obtain the time step information of the current diffusion process; The time step weighting coefficients are determined based on the time step information, where the time step weighting coefficients are negatively correlated with the time step information, so that the time step corresponding to the initial stage of denoising has a higher loss weight. The structural loss is adjusted by weighting coefficients based on the time step.

[0013] Secondly, embodiments of this application provide a diffusion model fine-tuning device, comprising: The data preprocessing module is used to obtain the training dataset of the target subject, encode the training samples in the training dataset to obtain the true latent space features, and construct noisy latent space features based on the true latent space features. The model inference module is used to load the pre-trained diffusion model, input the noisy latent space features into the pre-trained diffusion model for forward propagation, and obtain the predicted noise. The loss calculation module is used to determine the composite loss based on training samples, noisy latent space features and prediction noise. The composite loss includes basic reconstruction loss, structural loss based on normalized spatial attention weighting, and edge consistency loss based on channel-level grouped edge detection. The parameter update module is used to update the parameters of the pre-trained diffusion model based on the composite loss until the model converges, resulting in a fine-tuned diffusion model.

[0014] Thirdly, embodiments of this application provide a diffusion model fine-tuning device, which includes a processor, a memory, and a diffusion model fine-tuning program stored in the memory and executable by the processor, wherein when the diffusion model fine-tuning program is executed by the processor, it implements the steps of the diffusion model fine-tuning method as described above.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium storing a diffusion model fine-tuning program, wherein when the diffusion model fine-tuning program is executed by a processor, it implements the steps of the diffusion model fine-tuning method as described above.

[0016] The beneficial effects of the technical solutions provided in this application include: This application constructs a composite loss mechanism, comprising a basic reconstruction loss, a structure loss based on normalized spatial attention weighting, and an edge consistency loss based on channel-level grouping edge detection. This effectively addresses the issues of low structural fidelity and blurred details in existing technologies. Specifically, the structure loss based on normalized spatial attention weighting enables the model to focus on key structural regions of the target subject during training, improving the preservation of subject features and ensuring structural consistency of the generated results. By introducing the edge consistency loss based on channel-level grouping edge detection, the high-frequency edge information of latent space features is directly constrained, significantly enhancing the edge clarity and detail texture of the generated image. Furthermore, by combining the basic reconstruction loss, structure loss, and edge consistency loss through the composite loss mechanism, model parameter updates under multiple constraints are achieved, resulting in a high-fidelity diffusion model. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating an embodiment of the diffusion model fine-tuning method of this application; Figure 2 This is a schematic diagram of the calculation process for structural loss according to an embodiment of this application; Figure 3 For this application Figure 2 A detailed flowchart of step S201; Figure 4 This is a schematic diagram illustrating the process of calculating edge consistency loss according to an embodiment of this application; Figure 5 For this application Figure 4 A detailed flowchart of step S402; Figure 6 This is a schematic diagram illustrating the process of calculating composite loss according to an embodiment of this application; Figure 7 This is a schematic diagram illustrating the process of dynamically adjusting the loss weight according to an embodiment of this application; Figure 8 This is a functional module diagram of an embodiment of the diffusion model fine-tuning device of this application. Figure 9 This is a schematic diagram of the hardware structure of the diffusion model fine-tuning device involved in the embodiments of this application. Detailed Implementation

[0018] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0020] In a first aspect, embodiments of this application provide a method for fine-tuning a diffusion model.

[0021] In one embodiment, reference is made to Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the diffusion model fine-tuning method of this application. Figure 1 As shown, the methods for fine-tuning the diffusion model include: S101. Obtain the training dataset of the target subject, encode the training samples in the training dataset to obtain the true latent space features, and construct noisy latent space features based on the true latent space features. S102. Load the pre-trained diffusion model, input the noisy latent space features into the pre-trained diffusion model for forward propagation, and obtain the predicted noise. S103. Determine the composite loss based on the training samples, noisy latent space features and predicted noise. The composite loss includes the basic reconstruction loss, the structural loss based on normalized spatial attention weighting and the edge consistency loss based on channel-level grouping edge detection. S104. Update the parameters of the pre-trained diffusion model based on the composite loss until the model converges, and obtain the fine-tuned diffusion model.

[0022] Specifically, in the embodiment of the diffusion model fine-tuning method, in order to address the problems of easy distortion of the main structure and blurred edge details in the vertical domain fine-tuning process of the existing diffusion model, the embodiment designs a composite loss consisting of basic reconstruction loss, structural loss based on normalized spatial attention weighting, and edge consistency loss based on channel-level grouped edge detection to fine-tune the parameters of the pre-trained diffusion model.

[0023] In this embodiment, the training dataset includes training samples with structured cue words added, and general category samples. The training samples with structured cue words annotate the product with a three-layer semantic structure of subject-component-scene, providing key area cue information for the diffusion model; the general category samples serve as a priori retention dataset to prevent the model from forgetting basic semantics.

[0024] During the fine-tuning process, for the training samples, the implementation first encodes them into the latent space to obtain the true latent space features, and then adds noise, such as Gaussian noise. The noisy latent space features are obtained. Then, the noisy latent space features are forward propagated using a pre-trained diffusion model to obtain the predicted noise.

[0025] In the embodiment, the basic reconstruction loss can be obtained by calculating the standard mean square error between the actual noise and the predicted noise.

[0026] Then, for the training samples, the embodiment constructs a Gaussian distribution mask of the target subject in the training samples in the latent space, and assigns weights based on the Gaussian distribution mask, making the weight value of the target subject higher than that of the background region, thus obtaining a spatial attention weight map. Next, the spatial attention weight map is normalized and multiplied element-wise with the feature-by-feature tensor corresponding to the basic reconstruction loss to obtain the weighted structural loss. The normalization process ensures that the expected magnitude of the weighted structural loss is consistent with the global standard mean square error. Through this structural loss, the embodiment can force the consistency of the latent space structural boundaries, solving the structural distortion problem existing in current technologies when dealing with complex structures.

[0027] Next, based on the predicted noise, the embodiment inversely derives the predicted denoised latent space features from the noisy latent space features. Then, edge detection is performed on both the predicted denoised latent space features and the true latent space features to identify the predicted channel gradient magnitude and the true channel gradient magnitude. Specifically, in the edge detection process, the embodiment extends the Sobel operator along the channel dimension. Under each channel, depthwise separable convolutions are performed on the X and Y axes based on the Sobel operator to obtain the X-axis gradient magnitude component and the Y-axis gradient magnitude component for each channel. Then, the gradient magnitude components in the two directions are merged to obtain the predicted channel gradient magnitude and the true channel gradient magnitude for each channel.

[0028] Next, the embodiment calculates the channel-level grouped gradient difference between the predicted channel gradient magnitude and the true channel gradient magnitude for each channel. Then, it averages the channel-level grouped gradient differences for each channel (i.e., global averaging) to obtain the edge consistency loss. This embodiment uses this edge consistency loss to solve the problem that existing technologies often produce blurred high-frequency details and edge contours in generated images.

[0029] In this embodiment, the final loss function is a weighted composite loss consisting of the basic reconstruction loss, structural loss, and edge consistency loss. In each training step, the embodiment performs model inference only once, reusing the predicted noise and predicted denoising latent space features, and then calculates the three losses in parallel. This approach avoids the memory redundancy and state inconsistencies caused by multiple model calls in traditional loss calculation chains.

[0030] Furthermore, the embodiment introduces a time-step weighting mechanism. Specifically, the embodiment determines a time-step weighting coefficient based on the time-step information of the current diffusion process. This coefficient is negatively correlated with the time step, so that the time step in the initial stage of denoising has a higher loss weight. The embodiment adjusts the structural loss term in the composite loss according to this time-step weighting coefficient, strengthening the model's grasp of the overall geometric structure in the early stage of denoising.

[0031] During training, the model's performance is evaluated using multi-dimensional evaluation metrics, including at least one of structural similarity and semantic consistency metrics. For example, a comprehensive score combining image-text similarity, self-supervised feature distance, and perceptual quality is used. The training hyperparameters are dynamically adjusted based on this score, and the learning rate is automatically reduced in the later stages of training to converge to a local optimum. After the multi-dimensional evaluation metrics meet preset conditions or after reaching the required number of fine-tuning training epochs, the fine-tuned diffusion model is obtained.

[0032] In this embodiment, by constructing a composite loss consisting of a basic reconstruction loss, a structural loss based on normalized spatial attention weighting, and an edge consistency loss based on channel-level grouped edge detection, the problems of low structural fidelity and blurred details in existing technologies can be effectively solved. Specifically, by using the structural loss based on normalized spatial attention weighting, the model can focus on the key structural regions of the target subject during training, improving the ability to preserve subject features and ensuring the structural consistency of the generated results. By introducing the edge consistency loss based on channel-level grouped edge detection, the high-frequency edge information of latent space features is directly constrained, significantly enhancing the edge clarity and detail texture of the generated image. Furthermore, by combining the basic reconstruction loss, structural loss, and edge consistency loss through the composite loss, model parameter updates under multiple constraints are achieved, thereby obtaining a high-fidelity diffusion model.

[0033] Furthermore, in one embodiment, Figure 2 This is a schematic diagram of the calculation of structural loss according to an embodiment of this application, as shown below. Figure 2 As shown, the structural loss based on normalized spatial attention weighting is obtained in the following way: S201. Construct a spatial attention weight map of the training samples in the latent space; S202. Normalize the spatial attention weight map, and multiply the normalized spatial attention weight map element-wise with the feature-by-feature tensor corresponding to the basic reconstruction loss to obtain the weighted structural loss.

[0034] Furthermore, in one embodiment, Figure 3 For this application Figure 2 The detailed flowchart of step S201 is as follows: Figure 3As shown, a spatial attention weight map of the training samples is constructed in the latent space, including: S301. Construct a Gaussian distribution mask for the target subject in the training samples within the latent space; S302. Assign weights based on the Gaussian distribution mask, making the weight of the target area higher than that of the background area, to obtain the spatial attention weight map.

[0035] Specifically, in order to address the problem that the key structures of the main body in the generated results are easily deformed when the existing technology processes complex geometric structures, resulting in distortion of the results, the embodiment uses a normalized Gaussian spatial attention weighting mechanism to weight the basic reconstruction loss to obtain the structural loss, so as to automatically focus the learning focus of the model on the core area of ​​the main body.

[0036] In the embodiment, for training samples In this embodiment, the features are encoded into the latent space using a variational autoencoder to obtain the true latent space features. And randomly sample Gaussian noise Noisy latent space features are obtained by superimposing them in the latent space. Then, the implementation loads a pre-trained diffusion model, such as a Z-Image or FLUX pedestal model, to extract noisy latent space features. The pre-trained diffusion model is input for forward propagation to obtain the predicted noise. .

[0037] For basic reconstruction losses The example uses real noise and prediction noise The standard mean square error is calculated using the following formula:

[0038] in, This represents the mean square error function.

[0039] Then, based on the annotation information of the target subject in the training samples, the embodiment constructs a Gaussian distribution mask of the target subject in the latent space with the image center as the origin. Weights are assigned according to the Gaussian distribution mask, making the weight value of the target subject higher than that of the background region, resulting in a spatial attention weight map. Furthermore, to ensure that the expected magnitude of the structural loss is consistent with the global standard mean square error, the embodiment also needs to normalize this spatial attention weight map.

[0040] In this embodiment, the Gaussian mask is represented as:

[0041] in, Indicates at coordinate point The output weight value at that point, Represents the coordinates of the center point of the Gaussian distribution. The standard deviation is denoted as .

[0042] The normalization operation can be represented as:

[0043] in, and This represents the height and width of the global pixels.

[0044] The implementation example ensures that the expected magnitude of the weighted structural loss is consistent with the basic reconstruction loss through the above normalization operation.

[0045] Then, the implementation example performs element-wise multiplication between the normalized spatial attention weight map and the feature-wise tensor corresponding to the basic reconstruction loss to obtain the structural loss:

[0046] Through this structural loss, the embodiment assigns a higher gradient weight to the central region of the main body.

[0047] In this embodiment, a spatial attention weight map based on a Gaussian distribution mask is constructed in the latent space, making the weight of the target subject region significantly higher than that of the background region. Combined with normalization processing, this guides the model to adaptively focus on the key structural regions of the target subject during training. This approach enhances the gradient propagation efficiency of subject features, avoids fluctuations in the magnitude of loss caused by weight scaling, improves the integrity and fidelity of the subject structure in the generated result, and reduces subject deformation issues.

[0048] Furthermore, in one embodiment, Figure 4 This is a schematic diagram illustrating the process of calculating edge consistency loss in an embodiment of this application, as shown below. Figure 4 As shown, the edge consistency loss based on channel-level grouped edge detection is obtained in the following way: S401. Determine the predicted denoising latent space features based on the noisy latent space features and the predicted noise. S402. Perform edge detection on the predicted denoised latent space features and the real latent space features in each channel to obtain the gradient magnitude of the predicted channel and the gradient magnitude of the real channel. S403. Calculate the channel-level grouped gradient difference between the predicted channel gradient magnitude and the actual channel gradient magnitude, and perform a global average of the channel-level grouped gradient difference to obtain the edge consistency loss.

[0049] Furthermore, in one embodiment, Figure 5 For this application Figure 4 A detailed flowchart of step S402 is shown below. Figure 5As shown, edge detection is performed on the predicted denoised latent space features and the true latent space features in each channel to obtain the gradient magnitude of the predicted channel and the gradient magnitude of the true channel, including: S501. For the predicted denoised latent space features and the real latent space features, respectively, perform depthwise separable convolution on the X-axis and Y-axis based on the Sobel operator in each channel to obtain the corresponding X-axis gradient magnitude component and Y-axis gradient magnitude component for each channel. S502. Combine the X-axis gradient magnitude component and Y-axis gradient magnitude component of each corresponding channel to obtain the predicted channel gradient magnitude and the true channel gradient magnitude of each channel.

[0050] Specifically, in response to the problem that traditional fine-tuning methods lack constraints on edge contour information, which can easily lead to blurring of high-frequency details and edge contours, the embodiment designs an edge consistency loss based on channel-level grouped edge detection. This loss, by accurately capturing high-frequency structural information under multi-channel features, forces the prediction results to remain consistent with the target on the structural boundary.

[0051] In the embodiment, firstly based on the predicted noise From the noisy latent space features The predicted denoised latent space features are obtained by reverse calculation. The implementation then utilizes the Sobel operator to predict the denoised latent space features. and real latent space characteristics The gradient magnitude is calculated independently for each channel, and then globally averaged. Specifically, for latent space features, the system uses 3×3 Sobel kernels Gx and Gy to convolve each channel, calculates the gradient magnitudes in different directions, and then merges them to obtain the true edge response map for each channel. and predicted edge response map Then through comparison and The pixel-by-pixel differences are calculated, and the channel-level grouped gradient differences of each channel are averaged (i.e., global average) to obtain the edge consistency loss, which is expressed by the following formula:

[0052] in, Indicates the number of channels.

[0053] In this embodiment, by performing edge detection on each channel in the latent space and constraining the gradient magnitude difference, high-frequency edge information is directly enhanced, improving the edge clarity and detail texture of the generated image and effectively improving the problem of detail blurring.

[0054] Furthermore, in one embodiment, Figure 6 This is a schematic diagram of the process for calculating the composite loss according to an embodiment of this application, as shown below. Figure 6 As shown, the composite loss is determined based on the training samples, noisy latent space features, and prediction noise, including: S601. In each training step, the predicted noise and the predicted denoised latent space features are reused to compute the basic reconstruction loss, the structural loss based on normalized spatial attention weighting, and the edge consistency loss based on channel-level grouped edge detection in parallel. The predicted denoised latent space features are determined based on the noisy latent space features and the predicted noise. S602, the composite loss is obtained by merging the basic reconstruction loss, structural loss and edge consistency loss.

[0055] Specifically, in calculating the composite loss, unlike the traditional loss calculation chain, to avoid memory redundancy and state inconsistency caused by multiple model calls, this embodiment performs model inference only once in each standard training step to obtain the predicted noise and the corresponding predicted denoising latent space features, and then feeds them into each loss calculation module for "one source, multiple uses". The composite loss formula is expressed as:

[0056] in, , and These are the weights of each loss function.

[0057] This implementation not only improves training speed through this architecture, but also ensures that all constraints act on the exact same prediction distribution, thus avoiding gradient conflicts.

[0058] Furthermore, in one embodiment, Figure 7 This is a schematic diagram illustrating the process of dynamically adjusting the loss weight in an embodiment of this application, as shown below. Figure 7 As shown, the diffusion model fine-tuning method also includes: S701. Obtain the time step information of the current diffusion process; S702. Determine the time step weighting coefficients based on the time step information, wherein the time step weighting coefficients are negatively correlated with the time step information, so that the time step corresponding to the initial stage of denoising has a higher loss weight. S703. Adjust the structural loss according to the weighting coefficient of the time step.

[0059] Specifically, in order to enhance the model's grasp of the overall geometric structure in the early stages of denoising, the implementation example applies an additional loss penalty in the early large time steps and sets a time step weighting coefficient that is negatively correlated with the time step information.

[0060] Specifically, the implementation method is based on time step information. Introducing a time step reciprocal weighted term Then, the structural loss term is adjusted by weighting the reciprocal weighting term of the time step to strengthen the model's grasp of the overall geometric structure in the early stage of denoising, and further improve the integrity and restoration of the main structure in the generated result.

[0061] Furthermore, in some preferred embodiments, in order to balance instance learning and category generalization, the embodiments, in the selection of the dataset, in addition to the training samples of the target direction of model fine-tuning, also construct a prior retention dataset of general category samples to prevent the model from forgetting the basic semantics.

[0062] In a preferred embodiment, the fine-tuning workflow matrix can be divided into four stages: The first stage is exploration and baseline establishment, which involves rapid training with a small number of epochs using structured data and a basic LoRA injector, with r=8 and Alpha=16, to verify the quality of data cleaning and the effectiveness of prompt words. The second stage is hyperparameter search, in which the embodiment uses adaptive search and a composite loss engine to compare r=16 / 32 and different... The first stage involves combining time-step weighting to observe trajectory stability in the early denoising phase. The second stage is overfitting prevention fine-tuning, which injects prior samples from a general-purpose car into the model by preserving the sampler and using dynamic weight decay, monitoring the loss difference between the training and validation sets, and performing early stopping. The third stage is multimodal deployment, which uses an inference script generator and ComfyUI adapter to encapsulate the trained model into a standard interface and automatically generate an inference workflow that includes a dedicated sampler (such as AuraFlowShift).

[0063] During training, the implementation integrates three metrics—CLIP-I (image-text similarity), DINO (self-supervised feature distance), and LPIPS (perceptual quality)—to establish a comprehensive score. A cosine annealing strategy is employed to schedule the learning rate, automatically reducing it in the later stages of training to converge to a local optimum. Based on validation feedback, the LoRA Rank and Alpha coefficients are dynamically adjusted to achieve automated hyperparameter search.

[0064] Secondly, embodiments of this application also provide a diffusion model fine-tuning device.

[0065] In one embodiment, reference is made to Figure 8 , Figure 8 This is a functional module diagram of an embodiment of the diffusion model fine-tuning device of this application. Figure 8 As shown, the diffusion model fine-tuning device includes: The data preprocessing module 801 is used to obtain the training dataset of the target subject, encode the training samples in the training dataset to obtain the true latent space features, and construct noisy latent space features based on the true latent space features. The model inference module 802 is used to load the pre-trained diffusion model, input the noisy latent space features into the pre-trained diffusion model for forward propagation, and obtain the predicted noise. The loss calculation module 803 is used to determine the composite loss based on training samples, noisy latent space features and prediction noise. The composite loss includes basic reconstruction loss, structural loss based on normalized spatial attention weighting and edge consistency loss based on channel-level grouping edge detection. The parameter update module 804 is used to update the parameters of the pre-trained diffusion model based on the composite loss until the model converges, thus obtaining the fine-tuned diffusion model.

[0066] The functions of each module in the above-mentioned diffusion model fine-tuning device correspond to the steps in the above-mentioned diffusion model fine-tuning method embodiment, and their functions and implementation processes will not be described in detail here.

[0067] Thirdly, embodiments of this application provide a diffusion model fine-tuning device, which can be a personal computer (PC), laptop computer, server, or other device with data processing capabilities.

[0068] Reference Figure 9 , Figure 9 This is a schematic diagram of the hardware structure of the diffusion model fine-tuning device involved in the embodiments of this application. In the embodiments of this application, the diffusion model fine-tuning device may include a processor, a memory, a communication interface, and a communication bus.

[0069] The communication bus can be of any type and is used to interconnect the processor, memory, and communication interface.

[0070] The communication interface includes input / output (I / O) interfaces, physical interfaces, and logical interfaces used for interconnecting devices within the diffusion model fine-tuning device, as well as interfaces used for interconnecting the diffusion model fine-tuning device with other devices (such as other computing devices or user equipment). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user equipment can be displays, keyboards, etc.

[0071] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0072] The processor can be a general-purpose processor, which can call the diffusion model fine-tuning program stored in memory and execute the diffusion model fine-tuning method provided in the embodiments of this application. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the diffusion model fine-tuning program is called can be referred to in the various embodiments of the diffusion model fine-tuning method of this application, and will not be repeated here.

[0073] Those skilled in the art will understand that Figure 9 The hardware structure shown does not constitute a limitation of this application and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0074] Fourthly, embodiments of this application also provide a computer-readable storage medium.

[0075] The present application has a diffusion model fine-tuning program stored on a computer-readable storage medium, wherein when the diffusion model fine-tuning program is executed by a processor, it implements the steps of the diffusion model fine-tuning method as described above.

[0076] The method implemented when the diffusion model fine-tuning procedure is executed can be referred to in various embodiments of the diffusion model fine-tuning method of this application, and will not be repeated here.

[0077] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0078] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.

[0079] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.

[0080] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0081] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0082] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods of the various embodiments of this application.

[0083] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for fine-tuning a diffusion model, characterized in that, include: Obtain the training dataset of the target subject, encode the training samples in the training dataset to obtain the true latent space features, and construct noisy latent space features based on the true latent space features; Load the pre-trained diffusion model, input the noisy latent space features into the pre-trained diffusion model for forward propagation, and obtain the predicted noise; The composite loss is determined based on the training samples, the noisy latent space features, and the predicted noise. The composite loss includes a basic reconstruction loss, a structural loss based on normalized spatial attention weighting, and an edge consistency loss based on channel-level grouped edge detection. The parameters of the pre-trained diffusion model are updated based on the composite loss until the model converges, resulting in a fine-tuned diffusion model.

2. The diffusion model fine-tuning method according to claim 1, characterized in that, The structural loss based on normalized spatial attention weighting is obtained in the following way: Construct a spatial attention weight map of the training samples in the latent space; The spatial attention weight map is normalized, and the normalized spatial attention weight map is multiplied element-wise by the feature-by-feature tensor corresponding to the basic reconstruction loss to obtain the weighted structural loss.

3. The diffusion model fine-tuning method according to claim 2, characterized in that, The construction of the spatial attention weight map of the training samples in the latent space includes: Construct a Gaussian distribution mask for the target subject in the training samples within the latent space; Weights are assigned based on the Gaussian distribution mask, making the weight value of the target subject higher than that of the background region, thus obtaining a spatial attention weight map.

4. The diffusion model fine-tuning method according to claim 1, characterized in that, The edge consistency loss based on channel-level grouped edge detection is obtained in the following way: The predicted denoising latent space features are determined based on the noisy latent space features and the predicted noise. Edge detection is performed on the predicted denoised latent space features and the real latent space features in each channel to obtain the gradient magnitude of the predicted channel and the gradient magnitude of the real channel. The channel-level grouped gradient difference between the predicted channel gradient magnitude and the actual channel gradient magnitude is calculated, and the channel-level grouped gradient difference is globally averaged to obtain the edge consistency loss.

5. The diffusion model fine-tuning method according to claim 4, characterized in that, The step of performing edge detection on the predicted denoised latent space features and the real latent space features in each channel to obtain the predicted channel gradient magnitude and the real channel gradient magnitude includes: For the predicted denoised latent space features and the real latent space features, depthwise separable convolutions are performed on the X and Y axes based on the Sobel operator in each channel to obtain the X-axis gradient magnitude component and Y-axis gradient magnitude component of each channel. The X-axis gradient magnitude component and Y-axis gradient magnitude component of each corresponding channel are combined to obtain the predicted channel gradient magnitude and the true channel gradient magnitude of each channel.

6. The diffusion model fine-tuning method according to claim 1, characterized in that, The step of determining the composite loss based on the training samples, the noisy latent space features, and the predicted noise includes: In each training step, the predicted noise and the predicted denoised latent space features are reused to compute the basic reconstruction loss, the structure loss based on normalized spatial attention weighting, and the edge consistency loss based on channel-level grouped edge detection in parallel, wherein the predicted denoised latent space features are determined based on the noisy latent space features and the predicted noise. The composite loss is obtained by combining the basic reconstruction loss, the structural loss, and the edge consistency loss.

7. The diffusion model fine-tuning method according to claim 1, characterized in that, The method further includes: Obtain the time step information of the current diffusion process; The time step weighting coefficients are determined based on the time step information, wherein the time step weighting coefficients are negatively correlated with the time step information, so that the time step corresponding to the initial stage of denoising has a higher loss weight. The structural loss is weighted and adjusted according to the weighting coefficients of the time step.

8. A diffusion model fine-tuning device, characterized in that, include: The data preprocessing module is used to obtain the training dataset of the target subject, encode the training samples in the training dataset to obtain the true latent space features, and construct noisy latent space features based on the true latent space features. The model inference module is used to load the pre-trained diffusion model, input the noisy latent space features into the pre-trained diffusion model for forward propagation, and obtain the predicted noise. The loss calculation module is used to determine a composite loss based on the training samples, the noisy latent space features and the prediction noise. The composite loss includes a basic reconstruction loss, a structural loss based on normalized spatial attention weighting and an edge consistency loss based on channel-level grouped edge detection. The parameter update module is used to update the parameters of the pre-trained diffusion model based on the composite loss until the model converges, thus obtaining the fine-tuned diffusion model.

9. A diffusion model fine-tuning device, characterized in that, The diffusion model fine-tuning device includes a processor, a memory, and a diffusion model fine-tuning program stored in the memory and executable by the processor, wherein when the diffusion model fine-tuning program is executed by the processor, it implements the steps of the diffusion model fine-tuning method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a diffusion model fine-tuning program, wherein when the diffusion model fine-tuning program is executed by a processor, it implements the steps of the diffusion model fine-tuning method as described in any one of claims 1 to 7.

Images