Image generation training method for AIGC large model

Abstract

The present application relates to the technical field of image generation, and particularly relates to an image generation training method for an AIGC large model, comprising: obtaining standard images by preprocessing collected labeled images, injecting the standard images into an initial model for training to obtain a pre-training diffusion model, inputting random noise into the pre-training diffusion model, and generating initial images by reverse diffusion; optimizing the pre-training diffusion model according to the initial images to obtain an AIGC large model with image generation capability, processing image generation requirements by using the AIGC large model to generate target images; determining whether the number of reverse diffusion sampling steps needs to be increased; determining whether the sampling temperature of the image generation process needs to be increased; and determining the residual connection weight of deep features based on the attack iteration decay rate in the optimization process. The present application improves the generation stability of the target images.

Description

Technical Field

[0001] This invention relates to the field of image generation technology, and in particular to an image generation training method for AIGC large models. Background Technology

[0002] With artificial intelligence gradually becoming a core driver of digital content production, the demand for high-precision and controllable image generation is increasingly urgent in professional fields such as creative design, film and television production, and industrial simulation. Large-scale image generation models based on diffusion models and attention mechanisms are crucial because they can be trained by aligning with massive amounts of visual data through a generation paradigm that uses iterative noise reduction. This generates image content that conforms to semantic constraints and possesses visual plausibility, overcoming the limitations of traditional generation models in terms of resolution, diversity, and semantic consistency. This approach promotes clearer image generation and has significant industrial value and application potential. However, existing training methods still generally suffer from problems such as coarse text-image alignment granularity and weak controllability in detail generation, making it difficult to fully meet the high fidelity and high controllability requirements of professional scenarios for generated image results.

[0003] Chinese Patent Publication No. CN119863535A discloses a method, apparatus, and device for image generation and model training based on a large model. The method includes: extracting features from a foreground image corresponding to foreground content in an original image to obtain foreground features; performing image-text processing on a reference image to obtain global features and text features; the reference image is obtained based on the target style of the background content to be generated; and using a large model, based on the foreground features, the global features, and the text features, generating a target image including the foreground content and the background content. It is evident that the method, apparatus, and device for image generation and model training based on a large model suffer from insufficient stability in image generation due to unclear input feature differentiation weights and insufficient deep feature representation capabilities. Summary of the Invention

[0004] To address this issue, the present invention provides an image generation training method for large AIGC models, which overcomes the problem of insufficient stability in large model image generation caused by unclear input feature distinction weights and insufficient deep feature expression capabilities in existing technologies.

[0005] To achieve the above objectives, this invention provides an image generation and training method for large AIGC models, comprising:

[0006] The collected labeled images are preprocessed to obtain standard images. The standard images are then injected into the initial model for training to obtain a pre-trained diffusion model. Random noise is input into the pre-trained diffusion model, and the initial images are generated through back diffusion.

[0007] The pre-trained diffusion model is optimized based on the initial image to obtain a large AIGC model with image generation capabilities. The large AIGC model is then used to process the image generation requirements to generate the target image.

[0008] The gradient norm fluctuation amplitude of the back diffusion process per unit time is obtained, and the generation stability of the target image is determined based on the gradient norm fluctuation amplitude of the back diffusion process per unit time.

[0009] If the generation stability of the target image does not meet the requirements, determine whether it is necessary to increase the number of back-diffusion sampling steps;

[0010] If it is not necessary to increase the number of backdiffusion sampling steps, then obtain the Friesian distance between the target image and the standard image to determine whether the randomness of image generation meets the requirements;

[0011] If the randomness of the image generation does not meet the requirements, then determine whether it is necessary to increase the sampling temperature of the image generation process;

[0012] If it is not necessary to increase the sampling temperature of the image generation process, the residual connection weights of deep features are determined based on the attack iteration decay rate during the optimization process.

[0013] Furthermore, determining whether the generation stability of the target image meets the requirements based on the gradient norm fluctuation amplitude of the back-diffusion process includes:

[0014] The gradient norm fluctuation amplitude of the back diffusion process is compared with the preset first fluctuation amplitude.

[0015] If the gradient norm fluctuation amplitude of the back diffusion process is less than or equal to the preset first fluctuation amplitude, then the generation stability of the target image is determined to meet the requirements.

[0016] If the gradient norm fluctuation amplitude of the backdiffusion process is greater than the preset first fluctuation amplitude, it is determined that the generation stability of the target image does not meet the requirements, and it is determined whether the number of backdiffusion sampling steps needs to be increased based on the gradient norm fluctuation amplitude of the backdiffusion process.

[0017] Further, determine whether it is necessary to increase the number of backdiffusion sampling steps, including:

[0018] The gradient norm fluctuation amplitude of the reverse diffusion process is compared with the preset first fluctuation amplitude and the preset second fluctuation amplitude, respectively.

[0019] If the gradient norm fluctuation amplitude of the back diffusion process is greater than the preset second fluctuation amplitude, it is determined that the number of back diffusion sampling steps needs to be increased.

[0020] If the gradient norm fluctuation amplitude of the back diffusion process is greater than the preset first fluctuation amplitude and less than or equal to the preset second fluctuation amplitude, then it is determined that there is no need to increase the number of back diffusion sampling steps.

[0021] Furthermore, the increase in the number of back diffusion sampling steps is determined by the difference between the gradient norm fluctuation amplitude of the back diffusion process and the preset second fluctuation amplitude.

[0022] Furthermore, the randomness of image generation is determined based on the Fraser distance between the target image and the standard image to determine whether the randomness meets the requirements, including:

[0023] The Fraser distance between the target image and the standard image is compared with a preset second Fraser distance;

[0024] If the Frescher distance between the target image and the standard image is greater than the preset second Frescher distance, then it is determined that the randomness of image generation meets the requirements, and it is determined whether the number of back diffusion sampling steps meets the requirements.

[0025] If the Frescher distance between the target image and the standard image is less than or equal to the preset second Frescher distance, it is determined that the randomness of image generation does not meet the requirements, and it is determined whether the sampling temperature of the image generation process needs to be increased based on the Frescher distance between the target image and the standard image.

[0026] Further, determine whether it is necessary to increase the sampling temperature during the image generation process, including:

[0027] The Fraser distance between the target image and the standard image is compared with the preset first Fraser distance and the preset second Fraser distance, respectively.

[0028] If the Frescher distance between the target image and the standard image is greater than the preset first Frescher distance and less than or equal to the preset second Frescher distance, then it is determined that the sampling temperature of the image generation process needs to be increased.

[0029] If the Frescher distance between the target image and the standard image is less than or equal to the preset first Frescher distance, then it is determined that there is no need to increase the sampling temperature of the image generation process.

[0030] Furthermore, the increase in sampling temperature during the image generation process is determined by the difference between the preset second Fraser distance and the Fraser distance between the target image and the standard image.

[0031] Furthermore, the residual connection weights of deep features are determined based on the attack iteration decay rate during the optimization process, including:

[0032] The attack iteration decay rate during the optimization process is compared with the preset decay rate;

[0033] If the attack iteration decay rate during the optimization process is less than or equal to the preset decay rate, then the adversarial robustness of the image generation process is determined to meet the requirements, and it is not necessary to increase the residual connection weights of deep features. It is also determined whether the sampling temperature of the image generation process meets the requirements.

[0034] If the attack iteration decay rate during the optimization process is greater than the preset decay rate, it is determined that the adversarial robustness of the image generation process does not meet the requirements, and the residual connection weights of deep features need to be increased.

[0035] Furthermore, the attack iteration decay rate in the optimization process is the ratio of the relative change rate of the number of image generation errors caused by adversarial samples after multiple iterations to the number of iterations during the adversarial training process for image generation.

[0036] Furthermore, the increase in the residual connection weights of the deep features is determined by the difference between the attack iteration decay rate and the preset decay rate during the optimization process.

[0037] Compared with existing technologies, the beneficial effects of this invention are as follows: The method of this invention adjusts the number of backdiffusion sampling steps according to the gradient norm fluctuation amplitude of the backdiffusion process. Since excessively large single-step iteration update amplitude during the iterative sampling process of image generation amplifies random noise and errors, causing severe oscillations in the generation path and resulting in large fluctuations in the output image quality, increasing the number of backdiffusion sampling steps can limit the amplitude of single-step updates, increase the sampling frequency, and make the image generation more closely approximate the target distribution result in a more refined and controllable manner, reducing gradient norm fluctuations. The sampling temperature of the image generation process is adjusted according to the Fréchet distance between the target image and the standard image. Because the randomness of input noise is drastically attenuated under the set decision mechanism when it propagates through the generation model, small input noise fluctuations cannot affect the generation path, leading to... Noisy samples eventually converge to a finite number of patterns. By increasing the sampling temperature, the model can make more diverse decision responses to small changes in input noise. Image generation tends to select lower-probability but reasonable generation directions during sampling, providing diverse starting points for image generation and thus generating images with greater randomness. The residual connection weights of deep features are adjusted according to the attack iteration decay rate during the optimization process. Since the high-level semantic information contained in deep features is transmitted to subsequent layers with almost no loss, the image generation process is highly dependent on easily attacked semantic information, making the image generation quality extremely susceptible to interference. By increasing the residual connection weights of deep features, the output weight ratio of sensitive features is weakened, allowing the image to call more abstract features of each layer during the generation process, enhancing the anti-interference ability of the image generation process, and improving the generation stability of the target image.

[0038] Furthermore, the method of the present invention adjusts the number of backdiffusion sampling steps by setting a preset first fluctuation amplitude and a preset second fluctuation amplitude. Since an excessively large single-step iteration update amplitude will amplify random noise and error during the iterative sampling process of image generation, causing violent oscillations in the generation path and resulting in large fluctuations in the quality of the output image, by increasing the number of backdiffusion sampling steps, the amplitude of the single-step update can be limited, the sampling frequency can be increased, and the image generation can be made closer to the target distribution result in a more refined and controllable manner, reducing gradient norm fluctuations and further improving the generation stability of the target image.

[0039] Furthermore, the method of the present invention adjusts the sampling temperature of the image generation process by setting a preset first Friesian distance and a preset second Friesian distance. Since the randomness of the input noise is drastically attenuated under the set decision mechanism when the input noise propagates through the generation model, small fluctuations in input noise cannot affect the generation path, causing all noise samples to eventually converge to a finite number of patterns. By increasing the sampling temperature, the model makes more diverse decision responses to small changes in input noise. Image generation tends to choose a lower probability but reasonable generation direction during sampling, providing diverse starting points for image generation, thereby generating images with stronger randomness and further improving the generation stability of the target image.

[0040] Furthermore, the method of the present invention adjusts the residual connection weights of deep features by setting a preset attenuation rate. Since the high-level semantic information contained in deep features is transmitted to subsequent layers with almost no loss, the image generation process is highly dependent on the vulnerable semantic information, making the image generation quality extremely susceptible to interference. By increasing the residual connection weights of deep features, the output weight ratio of sensitive features is weakened, allowing the image to call more abstract features of each layer during the generation process, enhancing the anti-interference ability of the image generation process, and further improving the generation stability of the target image. Attached Figure Description

[0041] Figure 1 This is an overall flowchart of the image generation and training method for AIGC large models according to an embodiment of the present invention;

[0042] Figure 2 This is a flowchart illustrating the process of determining whether to increase the number of back-diffusion sampling steps in the image generation training method for the AIGC large model according to an embodiment of the present invention.

[0043] Figure 3 This is a flowchart illustrating the process of determining whether to increase the sampling temperature of the image generation process in an AIGC large model image generation training method according to an embodiment of the present invention.

[0044] Figure 4This is a flowchart illustrating the process of determining the residual connection weights of deep features in the image generation and training method for AIGC large models according to an embodiment of the present invention. Detailed Implementation

[0045] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0046] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0047] Please see Figure 1 As shown, it is an overall flowchart of the image generation and training method for AIGC large models in an embodiment of the present invention.

[0048] This invention provides an image generation and training method for large AIGC models, comprising:

[0049] Step S1: The collected labeled images are preprocessed to obtain standard images. The standard images are then injected into the initial model for training to obtain a pre-trained diffusion model. The initial images are generated using the pre-trained diffusion model based on the standard images.

[0050] Step S2: Optimize the pre-trained diffusion model based on the initial image to obtain a large AIGC model with image generation capabilities, and use the large AIGC model to process the image generation requirements to generate the target image;

[0051] Step S3: Obtain the gradient norm fluctuation amplitude of the back diffusion process per unit time, and determine whether the generation stability of the target image meets the requirements based on the gradient norm fluctuation amplitude of the back diffusion process per unit time.

[0052] Step S4: If the generation stability of the target image does not meet the requirements, determine whether it is necessary to increase the number of back diffusion sampling steps;

[0053] Step S5: If it is not necessary to increase the number of back diffusion sampling steps, then obtain the Friesian distance between the target image and the standard image to determine whether the randomness of image generation meets the requirements;

[0054] Step S6: If the randomness of the image generation does not meet the requirements, determine whether it is necessary to increase the sampling temperature of the image generation process;

[0055] Step S7: If it is not necessary to increase the sampling temperature of the image generation process, then determine the residual connection weights of deep features based on the attack iteration decay rate during the optimization process.

[0056] Specifically, the labeled images include CT scans with organ outlines, architectural floor plans with annotations, and parrot photos with species annotations.

[0057] Specifically, pretreatment includes cleaning, deweighting, adjusting size ratios, and normalization.

[0058] Specifically, a standard image is an image with a uniform format and fixed size that is output after the labeled image has been preprocessed.

[0059] Specifically, the process of injecting a standard image into an initial model for training to obtain a pre-trained diffusion model involves progressively adding noise to the standard image until it becomes pure noise, and then training the initial model to learn denoising through a reverse process to restore the standard image, thereby obtaining the pre-trained diffusion model.

[0060] Specifically, the initial model is an initial framework with basic image feature extraction capabilities and encoding and decoding structures.

[0061] Specifically, a pre-trained diffusion model is a diffusion model that has been trained from an initial model and is capable of recovering images from noise.

[0062] Specifically, the process of generating the initial image through backdiffusion starts with added random noise, gradually restores the clear image through multiple denoising iterations, and finally generates the initial image for training optimization.

[0063] Specifically, the process of optimizing the pre-trained diffusion model based on the initial image to obtain a large AIGC model with image generation capabilities involves calculating the loss function and performing adversarial training based on the distribution difference between the initial image and the standard image, continuously updating the parameters of the pre-trained diffusion model through gradient backpropagation, and gradually bringing the generated target image closer to the standard image in terms of visual quality and detail representation, ultimately resulting in a large AIGC model with image generation capabilities.

[0064] Specifically, image generation requirements include image style, image type, and image resolution.

[0065] Specifically, the number of backdiffusion sampling steps is the number of iterations performed to gradually denoise the added random noise.

[0066] Specifically, the sampling temperature during the image generation process is the core parameter that controls the scaling of the probability distribution when the model outputs the results. That is, the higher the sampling temperature during the image generation process, the smaller the probability difference between the generated target images, and the stronger the randomness of the target image generation.

[0067] Specifically, sampling is the process of selecting a specific output result from the probability distribution of the model's output results.

[0068] Specifically, the residual connection weights of deep features are parameters that measure the attenuation of deep features in the image during image generation through residual connections.

[0069] In implementation, the method of this invention adjusts the number of backdiffusion sampling steps based on the gradient norm fluctuation amplitude of the backdiffusion process. Because excessively large single-step iteration update amplitude during image generation amplifies random noise and errors, causing severe oscillations in the generation path and resulting in large fluctuations in output image quality, increasing the number of backdiffusion sampling steps limits the amplitude of single-step updates, increases the sampling frequency, and allows image generation to more closely approximate the target distribution in a more refined and controllable manner, reducing gradient norm fluctuations. The sampling temperature of the image generation process is adjusted based on the Fréchet distance between the target image and the standard image. Since the randomness of input noise is drastically attenuated under the established decision-making mechanism when propagating through the generation model, small input noise fluctuations cannot affect the generation path, leading to all noise samples ultimately... All converge to a finite number of patterns. By increasing the sampling temperature, the model can make more diverse decision responses to small changes in input noise. Image generation tends to choose a low-probability but reasonable generation direction during sampling, providing diverse starting points for image generation and thus generating images with stronger randomness. The residual connection weights of deep features are adjusted according to the attack iteration decay rate during the optimization process. Since the high-level semantic information contained in deep features is transmitted to subsequent layers with almost no loss, the image generation process is highly dependent on the easily attacked semantic information, making the image generation quality extremely susceptible to interference. By increasing the residual connection weights of deep features, the output weight ratio of sensitive features is weakened, allowing the image to call more abstract features of each layer during the generation process, enhancing the anti-interference ability of the image generation process, and improving the generation stability of the target image.

[0070] Specifically, determining whether the generated stability of the target image meets the requirements based on the gradient norm fluctuation amplitude of the back-diffusion process includes:

[0071] The gradient norm fluctuation amplitude of the back diffusion process is compared with the preset first fluctuation amplitude.

[0072] If the gradient norm fluctuation amplitude of the back diffusion process is less than or equal to the preset first fluctuation amplitude, then the generation stability of the target image is determined to meet the requirements.

[0073] If the gradient norm fluctuation amplitude of the back diffusion process is greater than the preset first fluctuation amplitude, then the generation stability of the target image is determined to be unsatisfactory.

[0074] The reasons why the generated stability of the target image may not meet the requirements could be that the randomness of image generation is not up to standard, or that the number of backdiffusion sampling steps is not up to standard. The next step is to determine which specific cause it is, which is also the process of determining whether to increase the number of backdiffusion sampling steps.

[0075] Specifically, determining whether the number of backdiffusion sampling steps needs to be increased includes:

[0076] The gradient norm fluctuation amplitude of the reverse diffusion process is compared with the preset first fluctuation amplitude and the preset second fluctuation amplitude, respectively.

[0077] If the gradient norm fluctuation amplitude of the back diffusion process is greater than the preset second fluctuation amplitude, it is determined that the number of back diffusion sampling steps needs to be increased.

[0078] If the gradient norm fluctuation amplitude of the back diffusion process is greater than the preset first fluctuation amplitude and less than or equal to the preset second fluctuation amplitude, then it is determined that there is no need to increase the number of back diffusion sampling steps.

[0079] Specifically, when the gradient norm fluctuation amplitude of the backdiffusion process exceeds a preset second fluctuation amplitude, it is determined that the reason for the unsatisfactory generation stability of the target image is that the number of backdiffusion sampling steps is insufficient, thus requiring an increase in the number of backdiffusion sampling steps. When the gradient norm fluctuation amplitude of the backdiffusion process exceeds a preset first fluctuation amplitude but is less than or equal to a preset second fluctuation amplitude, it can be preliminarily determined that the randomness of image generation is insufficient. The next step is to use the Fréchet distance between the target image and the standard image to make a final determination on whether the randomness of image generation meets the requirements, i.e., to determine whether the unsatisfactory generation stability of the target image is due to insufficient randomness in image generation.

[0080] It is understandable that the preset first fluctuation amplitude is smaller than the preset second fluctuation amplitude. The three intervals divided by the preset first and second fluctuation amplitudes correspond to three different scenarios:

[0081] The first interval is when the gradient norm fluctuation amplitude of the back diffusion process is less than or equal to the preset first fluctuation amplitude. The corresponding situation is: the generation stability of the target image meets the requirements, and no adjustment is needed.

[0082] The second interval is the gradient norm fluctuation amplitude of the back diffusion process is greater than the preset first fluctuation amplitude and less than or equal to the preset second fluctuation amplitude. The corresponding situation is: when the input noise propagates through the generation model, the randomness of the noise will be drastically attenuated under the set decision mechanism, so that the small input noise fluctuations cannot affect the generation path, causing all noise samples to eventually converge to a finite number of patterns. At this time, it is necessary to further judge whether the randomness of image generation meets the requirements.

[0083] The third interval is when the gradient norm fluctuation amplitude of the backdiffusion process is greater than the preset second fluctuation amplitude. The corresponding situation is: during the iterative sampling process of image generation, an excessively large single-step iterative update amplitude will amplify random noise and error, causing the generation path to oscillate violently, resulting in large fluctuations in the output image quality. At this time, it is necessary to adjust the number of backdiffusion sampling steps.

[0084] Understandably, in the process of AIGC large model image generation, using preset first fluctuation amplitude and preset second fluctuation amplitude to characterize the generation stability of the target image is essentially based on the hierarchical monitoring logic of gradient norm fluctuation and generation quality. This avoids the limitations of a single threshold in assessing the stability of the generation environment, and achieves hierarchical judgment and dynamic parameter adjustment control of the image generation process stability. The core function of the preset first fluctuation amplitude is to distinguish whether the generation stability of the target image meets the requirements; the core function of the preset second fluctuation amplitude is to distinguish the degree of inadequacy of the generation stability of the target image and the boundary for triggering adjustment. The preset first fluctuation amplitude and preset second fluctuation amplitude can be set according to actual working conditions. The setting of the preset first fluctuation amplitude and preset second fluctuation amplitude aims to ensure the generation stability and practicality of the target image. Optionally, the preset first fluctuation amplitude and preset second fluctuation amplitude are determined through a limited number of experiments by evaluating the image generation effect of different fluctuation amplitudes on the AIGC large model. The determined preset first fluctuation amplitude and preset second fluctuation amplitude should meet the requirement of being neither too small nor causing excessive interference to the image generation process of the AIGC large model. For example, the preset first fluctuation range is generally selected as [6%, 8%], and the preset second fluctuation range is generally selected as [9%, 11%].

[0085] Preferably, the first fluctuation range is 7% in the preferred embodiment, and the second fluctuation range is 10% in the preferred embodiment.

[0086] Specifically, the gradient norm fluctuation amplitude of the backdiffusion process per unit time is the difference between the maximum and minimum gradient norms of the parameter gradients of the pre-trained diffusion model in each backdiffusion process.

[0087] Specifically, the increase in the number of back diffusion sampling steps is determined by the difference between the gradient norm fluctuation amplitude of the back diffusion process and a preset second fluctuation amplitude.

[0088] Specifically, when the difference between the gradient norm fluctuation amplitude of the back diffusion process and the preset second fluctuation amplitude is within 2%, the number of back diffusion sampling steps is increased to 1.1 times the original value. When the difference between the gradient norm fluctuation amplitude of the back diffusion process and the preset second fluctuation amplitude exceeds 2%, the number of back diffusion sampling steps is increased by 10 steps for every 1% increase beyond the original value of 1.1 times. For example, when the difference between the gradient norm fluctuation amplitude of the back diffusion process and the preset second fluctuation amplitude is 4%, the current number of back diffusion sampling steps is 300 steps, and the increased number of back diffusion sampling steps is 300×1.1+10×2=350 steps.

[0089] In practice, the method of the present invention adjusts the backdiffusion sampling step size by setting a preset first fluctuation amplitude and a preset second fluctuation amplitude. Since an excessively large single-step iteration update amplitude will amplify random noise and error during the iterative sampling process of image generation, causing violent oscillations in the generation path and resulting in large fluctuations in the quality of the output image, by reducing the backdiffusion sampling step size, the amplitude of the single-step update can be limited, the sampling frequency can be increased, and the image generation can be made closer to the target distribution result in a more refined and controllable manner, reducing gradient norm fluctuations and further improving the generation stability of the target image.

[0090] Specifically, determining whether the randomness of image generation meets the requirements based on the Fréchet distance between the target image and the standard image includes:

[0091] The Fraser distance between the target image and the standard image is compared with a preset second Fraser distance;

[0092] If the Frescher distance between the target image and the standard image is greater than the preset second Frescher distance, then it is determined that the randomness of image generation meets the requirements, and it is determined whether the number of back diffusion sampling steps meets the requirements.

[0093] If the Frescher distance between the target image and the standard image is less than or equal to the preset second Frescher distance, it is determined that the randomness of image generation does not meet the requirements, and it is determined whether the sampling temperature of the image generation process needs to be increased based on the Frescher distance between the target image and the standard image.

[0094] Specifically, when the Fréchet distance between the target image and the standard image is greater than the second Fréchet distance, it is determined that the randomness of image generation meets the requirements. However, if the generation stability of the target image has not been determined to meet the requirements, it is necessary to further determine whether the number of backdiffusion sampling steps meets the requirements.

[0095] In practice, the number of back diffusion sampling steps is determined to meet the requirements by comparing the actual number of back diffusion sampling steps with the predetermined sampling step threshold. If the actual number of back diffusion sampling steps is less than the predetermined sampling step threshold, the number of back diffusion sampling steps is determined to be unacceptable. The predetermined sampling step threshold is the average number of back diffusion sampling steps monitored in the previous three months of the historical period.

[0096] If the number of backdiffusion sampling steps does not meet the requirements, the number of backdiffusion sampling steps is increased; if the number of backdiffusion sampling steps meets the requirements, the gradient norm fluctuation amplitude of the backdiffusion process is re-acquired, and the generation stability of the target image is re-evaluated.

[0097] When the Fréchet distance between the target image and the standard image is less than or equal to a preset second Fréchet distance, it can be determined that the reason for the target image's failure to meet generation stability requirements is that the randomness of image generation does not meet requirements. The reasons for this may include: the sampling temperature during image generation is not up to standard; or the robustness of the image generation process is not up to standard. The next step is to determine which specific cause it is, which involves deciding whether to increase the sampling temperature during image generation.

[0098] Specifically, determining whether the sampling temperature of the image generation process needs to be increased includes:

[0099] The Fraser distance between the target image and the standard image is compared with the preset first Fraser distance and the preset second Fraser distance, respectively.

[0100] If the Frescher distance between the target image and the standard image is greater than the preset first Frescher distance and less than or equal to the preset second Frescher distance, then it is determined that the sampling temperature of the image generation process needs to be increased.

[0101] If the Frescher distance between the target image and the standard image is less than or equal to the preset first Frescher distance, then it is determined that there is no need to increase the sampling temperature of the image generation process.

[0102] Specifically, when the Fréchet distance between the target image and the standard image is greater than a preset first Fréchet distance but less than or equal to a preset second Fréchet distance, it is determined that the reason for the unsatisfactory randomness of image generation is that the sampling temperature of the image generation process does not meet the requirements, thus requiring an increase in the sampling temperature of the image generation process. When the Fréchet distance between the target image and the standard image is less than or equal to the preset first Fréchet distance, it can be preliminarily determined that the adversarial robustness of the image generation process does not meet the requirements. The next step is to determine, based on the attack iteration decay rate during the optimization process, whether the adversarial robustness of the image generation process meets the requirements, i.e., to determine whether the reason for the unsatisfactory randomness of image generation is the unsatisfactory adversarial robustness of the image generation process.

[0103] It is understandable that the preset first Friesian distance is less than the preset second Friesian distance, and the three intervals divided by the preset first Friesian distance and the preset second Friesian distance correspond to three different cases:

[0104] The first interval is when the Frescher distance between the target image and the standard image is less than or equal to the preset first Frescher distance. The corresponding situation is: since the high-level semantic information contained in the deep features is transmitted to the subsequent layers with almost no loss, the image generation process is highly dependent on the semantic information that is easily attacked, which makes the image generation quality extremely easy to be interfered with. At this time, it is necessary to further determine whether the adversarial robustness of the image generation process meets the requirements.

[0105] The second interval is when the Fréchet distance between the target image and the standard image is greater than the preset first Fréchet distance and less than or equal to the preset second Fréchet distance. The corresponding situation is: when the input noise propagates through the generation model, the randomness of the noise will be drastically attenuated under the set decision mechanism, so that the small input noise fluctuations cannot affect the generation path, causing all noise samples to eventually converge to a finite number of patterns. At this time, it is necessary to adjust the sampling temperature of the image generation process.

[0106] The third interval is when the Fréchet distance between the target image and the standard image is greater than the preset second Fréchet distance. The corresponding situation is that the randomness of image generation meets the requirements. At this time, it is necessary to further determine whether the number of back diffusion sampling steps meets the requirements.

[0107] Understandably, in the process of AIGC large model image generation, using preset first and second Fraser distances to characterize the randomness of image generation is essentially a hierarchical evaluation logic based on the difference between the generated distribution and the target distribution. This overcomes the insufficient reliability of a single indicator in evaluating image diversity and achieves a quantitative grading of the model's generation quality. The core function of the preset first Fraser distance is to serve as an early warning line for triggering deep diagnostics of adversarial robustness; the core function of the preset second Fraser distance is to serve as a boundary for determining whether the generation diversity meets the standard. The preset first and second Fraser distances can be set according to actual working conditions. The setting of the preset first and second Fraser distances aims to ensure the stability and practicality of the generated target image. Optionally, the preset first and second Fraser distances are determined through a limited number of experiments by evaluating the effect of different Fraser distances on the image generation of the AIGC large model. The determined preset first and second Fraser distances should satisfy the condition that they are neither too small nor cause excessive interference to the image generation process of the AIGC large model. For example, the first Fraser distance is typically selected in the range of [19, 21], and the second Fraser distance is typically selected in the range of [29, 31].

[0108] Preferably, the first Fraser distance is preset to 20 in a preferred embodiment, and the second Fraser distance is preset to 30 in a preferred embodiment.

[0109] Specifically, the Frescher distance between the target image and the standard image is a parameter that quantizes the difference between the feature distribution of the generated image and the feature distribution of the standard image.

[0110] Specifically, the increase in the sampling temperature during the image generation process is determined by the difference between the preset second Fréchet distance and the Fréchet distance between the target image and the standard image.

[0111] Specifically, when the difference between the preset second Fréchet distance and the Fréchet distance between the target image and the standard image is within 5, the sampling temperature of the image generation process is increased to 1.1 times the original value. When the difference between the preset second Fréchet distance and the Fréchet distance between the target image and the standard image exceeds 5, the sampling temperature of the image generation process increases by 0.03 for every 1 unit above the original value, in addition to increasing to 1.1 times the original value. For example, when the difference between the preset second Fréchet distance and the Fréchet distance between the target image and the standard image is 7, the current sampling temperature of the image generation process is 0.8, and the increased sampling temperature of the image generation process is 0.8 × 1.1 + 0.03 × 2 = 0.94.

[0112] In practice, the method of this invention adjusts the sampling temperature of the image generation process by setting a preset first Friesian distance and a preset second Friesian distance. As the randomness of the input noise propagates through the generation model, it is drastically attenuated under the set decision mechanism, so that small fluctuations in input noise cannot affect the generation path. This causes all noise samples to eventually converge to a finite number of patterns. By increasing the sampling temperature, the model makes more diverse decision responses to small changes in input noise. During image generation, it tends to choose a generation direction with a lower probability but reasonableness, providing diverse starting points for image generation, thereby generating images with stronger randomness and further improving the generation stability of the target image.

[0113] Specifically, the residual connection weights of deep features are determined based on the attack iteration decay rate during the optimization process, including:

[0114] The attack iteration decay rate during the optimization process is compared with the preset decay rate;

[0115] If the attack iteration decay rate during the optimization process is less than or equal to the preset decay rate, then the adversarial robustness of the image generation process is determined to meet the requirements, and it is not necessary to increase the residual connection weights of deep features. It is also determined whether the sampling temperature of the image generation process meets the requirements.

[0116] If the attack iteration decay rate during the optimization process is greater than the preset decay rate, it is determined that the adversarial robustness of the image generation process does not meet the requirements, and the residual connection weights of deep features need to be increased.

[0117] Specifically, when the attack iteration decay rate during the optimization process is less than or equal to the preset decay rate, it is determined that the adversarial robustness of the image generation process meets the requirements. However, if it has been previously determined that the randomness of image generation does not meet the requirements, then it is necessary to further determine whether the sampling temperature of the image generation process meets the requirements.

[0118] In practice, the sampling temperature of the image generation process is compared with a predetermined sampling temperature threshold to determine whether the sampling temperature of the image generation process meets the requirements. If the sampling temperature of the image generation process is less than the predetermined sampling temperature threshold, the sampling temperature of the image generation process is determined to be unacceptable. The predetermined sampling temperature threshold is the average value of the sampling temperature of the image generation process monitored in the previous three months of the historical period.

[0119] If the sampling temperature during image generation does not meet the requirements, the sampling temperature during image generation is increased; if the sampling temperature during image generation meets the requirements, the Friesian distance between the target image and the standard image is re-acquired, and the randomness of image generation is re-evaluated.

[0120] When the attack iteration decay rate during the optimization process is greater than the preset decay rate, it can be determined that the reason why the randomness of image generation does not meet the requirements is that the adversarial robustness of the image generation process does not meet the requirements. Therefore, it is necessary to increase the residual connection weight of deep features.

[0121] It is understandable that the two intervals defined by the preset attenuation rate correspond to two different scenarios:

[0122] The first interval is when the attack iteration decay rate during the optimization process is less than or equal to the preset decay rate. The corresponding situation is: the adversarial robustness of the image generation process is determined to meet the requirements. At this time, it is necessary to further determine whether the sampling temperature of the image generation process meets the requirements.

[0123] The second interval is when the attack iteration decay rate during the optimization process is greater than the preset decay rate. The corresponding situation is that since the high-level semantic information contained in the deep features is transmitted to the subsequent layers with almost no loss, the image generation process is highly dependent on the easily attacked semantic information, which makes the image generation quality extremely easy to be interfered with. At this time, it is necessary to increase the residual connection weight of the deep features.

[0124] Understandably, in the AIGC large-scale model image generation process, using a preset decay rate to characterize the adversarial robustness of the image generation process is essentially based on the dynamic hierarchical monitoring logic of adversarial training. This achieves stability in the adoption of features within the model, meeting the requirements for adversarial robustness and generation quality stability in the high-standard image generation process. The core function of the preset decay rate is to serve as a dividing line for judging whether the adversarial robustness of the image generation process meets the requirements. The preset decay rate can be set according to actual working conditions. The setting of the preset decay rate aims to ensure the generation stability and usability of the target image. Optionally, the preset decay rate is determined through a limited number of experiments by evaluating the image generation effect of different attack iteration decay rates on the AIGC large-scale model. The determined preset decay rate should be neither too small nor cause excessive interference to the image generation process of the AIGC large-scale model. For example, the preset decay rate is generally selected in the range of [4% / attack, 6% / attack].

[0125] Preferably, the preferred embodiment of the preset attenuation rate is 5% / cycle.

[0126] Specifically, 5% / time represents a 5% reduction in the average number of image generation errors per iteration after multiple iterations.

[0127] Specifically, the attack iteration decay rate in the optimization process is the ratio of the relative change rate of the number of image generation errors caused by adversarial samples after multiple iterations to the number of iterations during the adversarial training process for image generation.

[0128] Specifically, after multiple iterations, the relative change rate of the number of image generation errors caused by adversarial examples is the ratio of the difference between the number of image generation errors before and after the iteration to the number of image generation errors before the iteration.

[0129] Specifically, the increase in the residual connection weights of the deep features is determined by the difference between the attack iteration decay rate and the preset decay rate during the optimization process.

[0130] Specifically, when the difference between the attack iteration decay rate and the preset decay rate during the optimization process is within 2% per iteration, the residual connection weight of the deep feature increases to 1.1 times the original value. When the difference between the attack iteration decay rate and the preset decay rate during the optimization process exceeds 2% per iteration, in addition to increasing to 1.1 times the original value, the residual connection weight of the deep feature increases by 0.03 for every 1% per iteration exceeding the original value. For example, when the difference between the attack iteration decay rate and the preset decay rate during the optimization process is 4% per iteration, the current residual connection weight of the deep feature is 0.5, and the increased residual connection weight of the deep feature is 0.5×1.1+0.03×2=0.61.

[0131] In practice, the method of the present invention adjusts the residual connection weights of deep features by setting a preset attenuation rate. Since the high-level semantic information contained in deep features is transmitted to subsequent layers with almost no loss, the image generation process is highly dependent on the vulnerable semantic information, which makes the image generation quality extremely easy to be interfered with. By increasing the residual connection weights of deep features, the output weight ratio of sensitive features is weakened, so that the image generation process calls more abstract features of each layer, enhances the anti-interference ability of the image generation process, and further improves the generation stability of the target image.

[0132] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. An image generation training method for an AIGC large model, characterized in that, The method comprises the steps of: standard images are obtained by preprocessing the collected labeled images, the standard images are injected into an initial model for training to obtain a pre-trained diffusion model, random noise is input into the pre-trained diffusion model, and an initial image is generated by reverse diffusion; an AIGC large model with image generation capability is obtained by optimizing the pre-trained diffusion model according to the initial image, and the AIGC large model is used to process image generation requirements to generate a target image; the gradient norm fluctuation amplitude of the reverse diffusion process within a unit time is obtained, and whether the generation stability of the target image meets the requirements is determined based on the gradient norm fluctuation amplitude of the reverse diffusion process within the unit time; if the generation stability of the target image does not meet the requirements, it is determined whether the reverse diffusion sampling step needs to be increased; if the reverse diffusion sampling step does not need to be increased, the Fréchet distance between the target image and the standard image is obtained to determine whether the randomness of image generation meets the requirements; if the randomness of image generation does not meet the requirements, it is determined whether the sampling temperature of the image generation process needs to be increased; if the sampling temperature of the image generation process does not need to be increased, the residual connection weight of the deep feature is determined based on the attack iteration decay rate in the optimization process.

2. The image generation training method for an AIGC large model according to claim 1, wherein, whether the generation stability of the target image meets the requirements is determined based on the gradient norm fluctuation amplitude of the reverse diffusion process, comprising: comparing the gradient norm fluctuation amplitude of the reverse diffusion process with a preset first fluctuation amplitude; if the gradient norm fluctuation amplitude of the reverse diffusion process is less than or equal to the preset first fluctuation amplitude, it is determined that the generation stability of the target image meets the requirements; if the gradient norm fluctuation amplitude of the reverse diffusion process is greater than the preset first fluctuation amplitude, it is determined that the generation stability of the target image does not meet the requirements, and whether the reverse diffusion sampling step needs to be increased is determined based on the gradient norm fluctuation amplitude of the reverse diffusion process.

3. The image generation training method for an AIGC large model according to claim 2, characterized in that, whether the reverse diffusion sampling step needs to be increased is determined, comprising: comparing the gradient norm fluctuation amplitude of the reverse diffusion process with the preset first fluctuation amplitude and a preset second fluctuation amplitude, respectively; if the gradient norm fluctuation amplitude of the reverse diffusion process is greater than the preset second fluctuation amplitude, it is determined that the reverse diffusion sampling step needs to be increased; if the gradient norm fluctuation amplitude of the reverse diffusion process is greater than the preset first fluctuation amplitude and less than or equal to the preset second fluctuation amplitude, it is determined that the reverse diffusion sampling step does not need to be increased.

4. The image generation training method for an AIGC large model according to claim 3, characterized in that, The increase amplitude of the reverse diffusion sampling step is determined by the difference between the gradient norm fluctuation amplitude of the reverse diffusion process and the preset second fluctuation amplitude.

5. The image generation training method for an AIGC large model according to claim 4, characterized in that, whether the randomness of image generation meets the requirements is determined based on the Fréchet distance between the target image and the standard image, comprising: comparing the Fréchet distance between the target image and the standard image with a preset second Fréchet distance; if the Fréchet distance between the target image and the standard image is greater than the preset second Fréchet distance, it is determined that the randomness of image generation meets the requirements, and whether the reverse diffusion sampling step meets the requirements is determined; If the Frechet distance between the target image and the standard image is less than or equal to the preset second Frechet distance, it is determined that the randomness of image generation does not meet the requirement, and it is determined whether the sampling temperature of the image generation process needs to be increased based on the Frechet distance between the target image and the standard image.

6. The image generation training method for an AIGC large model according to claim 5, characterized in that, The determination of whether the sampling temperature of the image generation process needs to be increased comprises: Comparing the Frechet distance between the target image and the standard image with a preset first Frechet distance and the preset second Frechet distance respectively; If the Frechet distance between the target image and the standard image is greater than the preset first Frechet distance and less than or equal to the preset second Frechet distance, it is determined that the sampling temperature of the image generation process needs to be increased; If the Frechet distance between the target image and the standard image is less than or equal to the preset first Frechet distance, it is determined that the sampling temperature of the image generation process does not need to be increased.

7. The image generation training method for an AIGC large model according to claim 6, characterized in that, The increase range of the sampling temperature of the image generation process is determined by the difference between the preset second Frechet distance and the Frechet distance between the target image and the standard image.

8. The image generation training method for an AIGC large model according to claim 7, characterized in that, The determination of the residual connection weight of the deep feature based on the attack iteration decay rate in the optimization process comprises: Comparing the attack iteration decay rate in the optimization process with a preset decay rate; If the attack iteration decay rate in the optimization process is less than or equal to the preset decay rate, it is determined that the adversarial robustness of the image generation process meets the requirement, the residual connection weight of the deep feature does not need to be increased, and it is determined whether the sampling temperature of the image generation process meets the requirement; If the attack iteration decay rate in the optimization process is greater than the preset decay rate, it is determined that the adversarial robustness of the image generation process does not meet the requirement, and the residual connection weight of the deep feature needs to be increased.

9. The image generation training method for an AIGC large model according to claim 8, characterized in that, The attack iteration decay rate in the optimization process is the ratio of the relative change rate of the number of times of image generation errors caused by the adversarial sample after multiple iterations to the number of iterations in the adversarial training process of image generation.

10. The image generation training method for an AIGC large model according to claim 9, characterized in that, The increase range of the residual connection weight of the deep feature is determined by the difference between the attack iteration decay rate in the optimization process and the preset decay rate.

Images