Image generation model training method and device, electronic equipment and storage medium

Abstract

The invention relates to a training method and device of an image generation model, electronic equipment and a storage medium. The method comprises the steps of obtaining a sample image, an image generation model comprising a noise adding network and a denoising network, and a to-be-trained consistency model; performing noise addition on the sample image based on a noise addition network and the current sampling time step to obtain a first sample noise image; denoising the first sample noise image based on a denoising network and a preset number of unit time intervals to obtain a second sample noise image; inputting the first sample noise image and the second sample noise image into a to-be-trained consistency model to obtain a first sample prediction image and a second sample prediction image; and training the to-be-trained consistency model according to the first sample prediction image, the second sample prediction image and target loss data determined by the dynamic weight in negative correlation with the current sampling time step to obtain a target image generation model. The model capable of directly generating the image can be obtained, and the image generation quality of the model can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a training method, device, electronic device and storage medium for an image generation model. Background Art

[0002] In recent years, image generation models such as generative adversarial networks and diffusion models have achieved remarkable results in various image generation tasks. In particular, diffusion models have attracted widespread attention due to their outstanding performance in generating high-quality images. However, traditional diffusion models usually require a large number of generation steps (usually between hundreds and thousands of steps), which leads to high computational costs and long generation time, limiting their widespread use in practical image generation tasks.

[0003] In the related art, many methods have been proposed to accelerate the generation process of diffusion models, such as improving the denoising step, introducing more efficient sampling strategies, samplers, etc. However, while these methods accelerate image generation, they often lead to a decrease in the quality of the generated images or an increase in the complexity of model training. Therefore, finding a solution that can significantly reduce the number of generation steps while maintaining or improving the quality of the generated images has become an important challenge in image generation tasks. Summary of the invention

[0004] The present disclosure provides a training method, device, electronic device and storage medium for an image generation model to at least solve the problems in the related art. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a method for training an image generation model is provided, comprising:

[0006] Acquire a sample image, a reference image generation model, and a consistency model to be trained, wherein the reference image generation model includes a reference denoising network and a reference denoising network;

[0007] Based on the reference denoising network and the current sampling time step corresponding to the reference denoising network, the sample image is subjected to denoising processing to obtain a first sample noise image; the current sampling time step includes at least one unit time interval;

[0008] Based on the reference denoising network and a preset number of unit time intervals, denoising the first sample noise image to obtain a second sample noise image; the preset number is not greater than the number of unit time intervals in the at least one unit time interval;

[0009] Inputting the first sample noise image into the consistency model to be trained, performing image generation processing, and obtaining a first sample prediction image;

[0010] Inputting the second sample noise image into the consistency model to be trained, performing image generation processing, and obtaining a second sample prediction image;

[0011] Determine a dynamic weight corresponding to the current sampling time step; the dynamic weight is negatively correlated with the current sampling time step;

[0012] Determining target loss data according to the first sample prediction image, the second sample prediction image and the dynamic weight;

[0013] The consistency model to be trained is trained according to the target loss data to obtain a target image generation model.

[0014] In an optional embodiment, determining the dynamic weight corresponding to the current sampling time step includes:

[0015] Obtaining a preset noise schedule corresponding to the reference noise adding network; the hyperparameters in the preset noise schedule indicate the noise increase intensity corresponding to different time steps; the noise increase intensity is negatively correlated with the unit time interval;

[0016] Determining at least one current hyperparameter corresponding to the at least one unit time interval from the preset noise schedule;

[0017] The at least one current hyperparameter is cumulatively multiplied to obtain the dynamic weight corresponding to the current sampling time step.

[0018] In an optional embodiment, determining target loss data according to the first sample prediction image, the second sample prediction image and the dynamic weight includes:

[0019] Determining original loss data according to the first sample prediction image and the second sample prediction image;

[0020] Based on the dynamic weight, the original loss data is adjusted to obtain the target loss data.

[0021] In an optional embodiment, the reference noisy network includes a first feature representation subnetwork, a noise sampling subnetwork and a noise addition subnetwork, and the sample image is subjected to noise processing based on the reference noisy network and the current sampling time step corresponding to the reference noisy network to obtain a first sample noise image, including:

[0022] Inputting the sample image into the feature representation subnetwork, performing feature representation processing, and obtaining original sample feature data; the original sample feature data is image feature data or latent space feature data;

[0023] Based on the noise sampling subnetwork, noise sampling processing is performed to obtain current noise sampling data;

[0024] Based on the current sampling time step, determining the current first weight corresponding to the original sample feature data and the current second weight corresponding to the current noise sampling data from the preset noise scheduling table corresponding to the reference noise adding network; the hyperparameters in the preset noise scheduling table are used to indicate the noise increase intensity corresponding to different time steps;

[0025] Input the original sample feature data, the current first weight, the current noise sampling data and the current second weight into the noise addition subnetwork to perform noise addition processing to obtain first sample feature data;

[0026] The first sample noise image is obtained according to the first sample characteristic data.

[0027] In an optional embodiment, the reference denoising network includes a second feature representation subnetwork, a noise prediction subnetwork and a noise removal subnetwork, and the denoising process is performed on the first sample noise image based on the reference denoising network and a preset number of unit time intervals to obtain a second sample noise image, including:

[0028] Inputting the first sample noise image into the second feature representation subnetwork, performing feature representation processing, and obtaining first sample feature data; the first sample feature data is image feature data or latent space feature data;

[0029] Inputting the first sample feature data and the preset number of unit time intervals into the noise prediction subnetwork, performing noise prediction processing, and obtaining current noise prediction data corresponding to the preset number of unit time intervals;

[0030] Inputting the first sample feature data and the current noise prediction data into the noise removal subnetwork to perform noise removal processing to obtain second sample feature data;

[0031] The second sample noise image is obtained according to the second sample characteristic data.

[0032] In an optional embodiment, the method further includes:

[0033] Get the prompt text;

[0034] The step of inputting the first sample noise image into the consistency model to be trained and performing image generation processing to obtain a first sample prediction image includes:

[0035] Inputting the first sample noise image and the prompt text into the consistency model to be trained, performing image generation processing with the prompt text as a constraint condition, and obtaining the first sample prediction image;

[0036] The step of inputting the second sample noise image into the consistency model to be trained and performing image generation processing to obtain a second sample prediction image includes:

[0037] The second sample noise image and the prompt text are input into the consistency model to be trained, and image generation processing is performed with the prompt text as a constraint condition to obtain the second sample predicted image.

[0038] In an optional embodiment, the target image generation model includes a target noise addition network and a target generation network, and the method further includes:

[0039] Get the image to be processed;

[0040] The image to be processed is input into the target image generation model, and image generation processing is performed a preset number of times in the target image generation model to obtain a target generated image corresponding to the image to be processed; wherein, during any one of the image generation processings, the following steps are performed:

[0041] Based on the target noise adding network and the current random time step, performing noise adding processing on the current image to be processed to obtain a current noisy image;

[0042] Inputting the current noise image into the target generation network to perform image generation processing to obtain a current candidate generated image;

[0043] Among them, in the first image generation processing process, the current image to be processed is the image to be processed; in the non-first image generation processing process, the current image to be processed is the current candidate generation image obtained in the image generation processing round before the current image generation processing round.

[0044] According to a second aspect of an embodiment of the present disclosure, there is provided a training device for an image generation model, comprising:

[0045] An acquisition module is configured to acquire a sample image, a reference image generation model, and a consistency model to be trained, wherein the reference image generation model includes a reference denoising network and a reference denoising network;

[0046] A denoising module is configured to perform denoising processing on the sample image based on the reference denoising network and a current sampling time step corresponding to the reference denoising network to obtain a first sample noise image; the current sampling time step includes at least one unit time interval;

[0047] a denoising module configured to perform denoising processing on the first sample noise image based on the reference denoising network and a preset number of unit time intervals to obtain a second sample noise image; the preset number is not greater than the number of unit time intervals in the at least one unit time interval;

[0048] A first prediction module is configured to input the first sample noise image into the consistency model to be trained, perform image generation processing, and obtain a first sample prediction image;

[0049] A second prediction module is configured to input the second sample noise image into the consistency model to be trained, perform image generation processing, and obtain a second sample prediction image;

[0050] A weight determination module is configured to determine a dynamic weight corresponding to the current sampling time step; the dynamic weight is negatively correlated with the current sampling time step;

[0051] a target loss calculation module, configured to determine target loss data according to the first sample prediction image, the second sample prediction image and the dynamic weight;

[0052] The training module is configured to train the consistency model to be trained according to the target loss data to obtain a target image generation model.

[0053] In an optional embodiment, the weight determination module includes:

[0054] A first acquisition unit is configured to execute acquisition of a preset noise schedule corresponding to the reference noise adding network; the hyperparameters in the preset noise schedule indicate the noise increase intensity corresponding to different time steps; the noise increase intensity is negatively correlated with the unit time interval;

[0055] A hyperparameter determination unit, configured to determine at least one current hyperparameter corresponding to the at least one unit time interval from the preset noise schedule;

[0056] The dynamic weight calculation unit is configured to perform cumulative multiplication of the at least one current hyperparameter to obtain the dynamic weight corresponding to the current sampling time step.

[0057] In an optional embodiment, the target loss calculation module includes:

[0058] an original loss data determining unit, configured to determine original loss data according to the first sample prediction image and the second sample prediction image;

[0059] The target loss data determination unit is configured to adjust the original loss data based on the dynamic weight to obtain the target loss data.

[0060] In an optional embodiment, the reference noise adding network includes a first feature representation subnetwork, a noise sampling subnetwork and a noise adding subnetwork, and the noise adding module includes:

[0061] A first feature representation unit is configured to input the sample image into the first feature representation subnetwork, perform feature representation processing, and obtain original sample feature data; the original sample feature data is image feature data or latent space feature data;

[0062] A noise sampling unit is configured to perform noise sampling processing based on the noise sampling subnetwork to obtain current noise sampling data;

[0063] A noise weight determination unit is configured to determine, based on the current sampling time step, a current first weight corresponding to the original sample feature data and a current second weight corresponding to the current noise sampling data from a preset noise scheduling table corresponding to the reference noise adding network; the hyperparameters in the preset noise scheduling table are used to indicate the noise increase intensity corresponding to different time steps;

[0064] A noise adding unit is configured to input the original sample feature data, the current first weight, the current noise sampling data and the current second weight into the noise adding sub-network, perform noise adding processing, and obtain the first sample feature data;

[0065] The first sample noise image determining unit is configured to obtain the first sample noise image according to the first sample feature data.

[0066] In an optional embodiment, the reference denoising network includes a second feature representation subnetwork, a noise prediction subnetwork and a noise removal subnetwork, and the denoising module includes:

[0067] A second feature representation unit is configured to input the first sample noise image into the second feature representation subnetwork, perform feature representation processing, and obtain first sample feature data; the first sample feature data is image feature data or latent space feature data;

[0068] A noise prediction unit is configured to input the first sample feature data and the preset number of unit time intervals into the noise prediction subnetwork, perform noise prediction processing, and obtain current noise prediction data corresponding to the preset number of unit time intervals;

[0069] a noise removal unit configured to input the first sample feature data and the current noise prediction data into the noise removal subnetwork, perform noise removal processing, and obtain second sample feature data;

[0070] The second sample noise image determining unit is configured to obtain the second sample noise image according to the second sample feature data.

[0071] In an optional embodiment, the device further comprises:

[0072] A second acquisition unit is configured to execute acquisition of prompt text;

[0073] The first prediction module comprises:

[0074] A first condition generating unit is configured to input the first sample noise image and the prompt text into the consistency model to be trained, perform image generation processing with the prompt text as a constraint condition, and obtain the first sample prediction image;

[0075] The second prediction module comprises:

[0076] The first condition generating unit is configured to execute the image generation processing with the prompt text as a constraint condition on the second sample noise image and the prompt text, so as to obtain the second sample predicted image.

[0077] In an optional embodiment, the target image generation model includes a target noise addition network and a target generation network, and the device further includes:

[0078] A third acquisition unit is configured to acquire an image to be processed;

[0079] The image generation processing unit is configured to input the image to be processed into the target image generation model, perform a preset number of image generation processes in the target image generation model, and obtain a target generated image corresponding to the image to be processed; wherein, during any one of the image generation processes, the following steps are performed:

[0080] A random noise adding subunit is configured to perform noise adding processing on the current image to be processed based on the target noise adding network and the current random time step to obtain a current noisy image;

[0081] An image generation subunit is configured to input the current noise image into the target generation network, perform image generation processing, and obtain a current candidate generated image;

[0082] Among them, in the first image generation processing process, the current image to be processed is the image to be processed; in the non-first image generation processing process, the current image to be processed is the current candidate generation image obtained in the image generation processing round before the current image generation processing round.

[0083] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the training method of the image generation model described in any one of the first aspect of the embodiments of the present disclosure.

[0084] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform a training method for an image generation model as described in any one of the first aspect of the embodiment of the present disclosure.

[0085] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement a training method for an image generation model as described in any one of the first aspect of an embodiment of the present disclosure.

[0086] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0087] The present disclosure first performs a noise addition process on the acquired sample image based on a reference denoising network in a reference image generation model and a current sampling time step corresponding to the reference denoising network to obtain a first sample noise image, wherein the current sampling time step includes at least one unit time interval; performs a denoising process on the first sample noise image based on a reference denoising network in the reference image generation model and a preset number of unit time intervals to obtain a second sample noise image, wherein the preset number is not greater than the number of unit time intervals included in the current sampling time step; then inputs the first sample noise image into a consistency model to be trained, performs an image generation process to obtain a first sample prediction image, and inputs the second sample noise image into the consistency model to be trained, performs an image generation process to obtain a second sample prediction image; in the process of determining the loss data, a dynamic weight that is negatively correlated with the current sampling time step is introduced, and the consistency model to be trained is trained using the target loss data determined based on the first sample prediction image, the second sample prediction image and the dynamic weight, so as to obtain a target image generation model with the ability to directly generate images.

[0088] In the technical solution provided by the embodiment of the present disclosure, the reference image generation model is used to perform distillation training on the consistency model to be trained, so as to learn the image generation ability of the reference image generation model, and the amount of training can be reduced by sampling the time step, thereby improving the training efficiency; in the technical solution provided by the embodiment of the present disclosure, the target loss data is calculated based on the self-consistency characteristics that the consistency model to be trained needs to meet, and the consistency model to be trained is trained using the target loss data. The obtained target image generation model can have the ability to directly generate images within a single unit time interval (i.e., a single step), which greatly reduces the number of steps for image generation, and can greatly save inference resources and reduce performance requirements in the application stage of image generation; in the technical solution provided by the embodiment of the present disclosure, the target loss data introduces a dynamic weight that is negatively correlated with the current sampling time step, i.e., the dynamic weight decreases with the increase of the current sampling time step, which can enhance the attention of the consistency model to be trained to the details and texture of the low-noise image when the current sampling time step is relatively small, thereby improving the image generation quality of the target image generation model.

[0089] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0091] Figure 1 is a schematic diagram of an application environment of an image generation model according to an exemplary embodiment;

[0092] Figure 2 is a flowchart of a training method for an image generation model according to an exemplary embodiment;

[0093] Figure 3 is a schematic diagram of a multi-step noise addition process according to an exemplary embodiment;

[0094] Figure 4 is a schematic diagram of a single-step denoising process according to an exemplary embodiment;

[0095] Figure 5 is a schematic diagram of a process of determining a dynamic weight according to an exemplary embodiment;

[0096] Figure 6 is a flowchart of an application method of an image generation model according to an exemplary embodiment;

[0097] Figure 7is a block diagram of a training device for an image generation model according to an exemplary embodiment;

[0098] Figure 8 is a block diagram of an electronic device for implementing a training method for an image generation model according to an exemplary embodiment;

[0099] Fig. 9 It is a block diagram of another electronic device for implementing a training method for an image generation model according to an exemplary embodiment. DETAILED DESCRIPTION

[0100] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0101] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0102] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0103] See also Figure 1 , which is a schematic diagram of an application environment of an image generation model according to an exemplary embodiment, the application environment may include a terminal 110 and a server 120, and the terminal 110 and the server 120 may be connected via a wired network or a wireless network.

[0104] The terminal 110 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc., but is not limited thereto. An application (Application, referred to as App) may be installed in the terminal 110. The application may be an independent application or a subprogram in an independent application. The user of the terminal 110 may log in to the application through pre-registered user information, which may include an account number and a password. The server 120 may be a server that provides background services for the application in the terminal 110, or may be other servers that are connected and communicated with the background server of the application. It may be a single server or a server cluster composed of multiple servers.

[0105] In one embodiment of the present disclosure, the terminal 110 may send an image generation request for an image to be processed to the server 120 in response to the user's image generation operation. The server 120 receives the image generation request, performs image generation processing according to the image generation request, and returns a target generated image corresponding to the image to be processed to the terminal 110. In a specific application, the target generated image may be a change in the style, content, etc. of the image to be processed. Specifically, the server 120 runs a target image generation model, which includes a target noise addition network and a target generation network. In response to the image generation request, the server 120 inputs the image to be processed into the target image generation model, performs a preset number of image generation processes in the target image generation model, and obtains a target generated image corresponding to the image to be processed; wherein, in any image generation process, the following steps are performed: based on the target noisy network and the current random time step, the current image to be processed is subjected to noise processing to obtain the current noise image; the current noise image is input into the target generation network, and image generation processing is performed to obtain the current candidate generated image; wherein, in the first image generation process, the current image to be processed is the image to be processed; in the non-first image generation process, the current image to be processed is the current candidate generated image obtained in the image generation process round before the current image generation process round. Under the premise that the target image generation model has the ability to directly generate images in one step, by appropriately increasing the number of steps and adopting multi-step generation, the image generation speed and image generation quality can be balanced.

[0106] The target image generation model is obtained by the following training method: obtaining a sample image, a reference image generation model and a consistency model to be trained, wherein the reference image generation model includes a reference denoising network and a reference denoising network; based on the reference denoising network and the current sampling time step corresponding to the reference denoising network, the sample image is denoised to obtain a first sample noise image; the current sampling time step includes at least one unit time interval; based on the reference denoising network and a preset number of unit time intervals, the first sample noise image is denoised to obtain a second sample noise image; the preset number is not greater than the number of unit time intervals in at least one unit time interval; the first sample noise image is input into the consistency model to be trained, image generation processing is performed to obtain a first sample prediction image; the second sample noise image is input into the consistency model to be trained, image generation processing is performed to obtain a second sample prediction image; the dynamic weight corresponding to the current sampling time step is determined; the dynamic weight is negatively correlated with the current sampling time step; the target loss data is determined according to the first sample prediction image, the second sample prediction image and the dynamic weight; the consistency model to be trained is trained according to the target loss data to obtain the target image generation model. In the above training method, the reference image generation model is used to perform distillation training on the consistency model to be trained, so as to learn the image generation ability of the reference image generation model, and the amount of training can be reduced by sampling the time step, thereby improving the training efficiency; in the above training method, the target loss data is calculated based on the self-consistency characteristics that the consistency model to be trained needs to meet, and the consistency model to be trained is trained using the target loss data. The obtained target image generation model can have the ability to directly generate images within a single unit time interval (i.e., a single step), which greatly reduces the number of steps for image generation, and can greatly save inference resources and reduce performance requirements in the application stage of image generation; in the above training method, the target loss data introduces a dynamic weight that is negatively correlated with the current sampling time step, i.e., the dynamic weight decreases as the current sampling time step increases, which can enhance the attention of the consistency model to be trained to the details and texture of the image when the current sampling time step is relatively small, thereby improving the image generation quality of the target image generation model.

[0107] In another embodiment of the present disclosure, the terminal 110 may also run a target image generation model, so that the terminal 110 can directly respond to the user's image generation operation and generate a target generation image corresponding to the image to be processed.

[0108] In another embodiment of the present disclosure, the terminal 110 can generate an image generation request for the image to be processed and the prompt text in response to the user's image generation operation, so that the target image generation model running on the terminal 110 responds to the image production request, performs image generation processing on the image to be processed with the prompt text as a constraint condition, and obtains the target generated image corresponding to the image to be processed and the prompt text. In specific applications, the prompt text can indicate the style, content and other attributes of the generated image.

[0109] Figure 2 is a flowchart of a method for training an image generation model according to an exemplary embodiment. Figure 2 As shown, the following steps may be included:

[0110] In step S210, a sample image, a reference image generation model and a consistency model to be trained are obtained, wherein the reference image generation model includes a reference denoising network and a reference denoising network.

[0111] In the disclosed embodiment, the reference image generation model can be an image generation model trained based on a diffusion model, including a reference denoising network and a reference denoising network, by forward denoising an image to Gaussian noise, and then reverse denoising from Gaussian noise to achieve image denoising, and finally achieve image generation. Traditional diffusion models usually require a large number of generation steps (usually between hundreds and thousands of steps), which leads to high computational cost and long generation time, limiting their widespread use in actual image generation tasks.

[0112] In the disclosed embodiment, a reference image generation model is used to perform distillation training on a consistency model to be trained, so as to quickly learn the image generation capability of the reference image generation model.

[0113] In the disclosed embodiment, the consistency model to be trained can be a deep neural network model of any form and is constrained by a consistency function so that the target image generation model obtained after training is a consistency model with self-consistency features. The consistency features possessed by the consistency model can refer to the description of the subsequent embodiments and will not be described here. The disclosed embodiment performs consistency distillation training on the consistency model to be trained, so that the target image generation model not only learns the image generation capability of the reference image generation model, but also has the ability to directly generate images in a single step, thereby reducing the number of generation steps and accelerating the generation process.

[0114] In step S220, based on the reference noisy network and the current sampling time step corresponding to the reference noisy network, the sample image is subjected to noise processing to obtain a first sample noise image; the current sampling time step includes at least one unit time interval.

[0115] In the disclosed embodiment, the reference noise adding network is set to be able to implement noise adding processing with a maximum time step (timestep) of T (corresponding to T unit time intervals). In the current training round, a time step t can be randomly sampled from 1 to T as the current sampling time step (including t unit time intervals, t≥1), so as to perform noise adding processing on the sample image based on the current sampling time step to obtain the first sample noise image corresponding to the current sampling time step t. By dynamically sampling the time step, the training rounds and training amount can be effectively reduced, thereby improving the training efficiency.

[0116] It should be noted that the current sampling time step t includes t unit time intervals, which is a discretized representation of the time domain of the noise addition process.

[0117] In one embodiment of the present disclosure, the sample image is subjected to noise processing based on the current sampling time step, which can be specifically implemented as t times of random noise sampling and addition. In order to improve the noise adding efficiency, the noise adding process can also be controlled by using a preset noise scheduler, so that only one random noise sampling and addition is required to achieve the effect of performing t steps of noise adding processing on the sample image.

[0118] Specifically, Figure 3 As shown, the reference noise adding network includes a first feature representation subnetwork, a noise sampling subnetwork and a noise adding subnetwork. Step S220 may include:

[0119] In step S221, the sample image is input into the first feature representation subnetwork, and feature representation processing is performed to obtain original sample feature data; the original sample feature data is image feature data or latent space feature data.

[0120] In a specific embodiment, the sample image is input into the first feature representation subnetwork, and the original sample feature data corresponding to the sample image can be obtained through specific image feature extraction and embedding representation. When the reference image generation model generates images in the image pixel space, the original sample feature data is the image feature data; when the reference image generation model generates images in a low-dimensional latent space, the original sample feature data is the latent space feature data. It can be understood that image generation in a low-dimensional latent space can reduce resource consumption and accelerate model training.

[0121] In step S222, noise sampling processing is performed based on the noise sampling subnetwork to obtain current noise sampling data.

[0122] In a specific embodiment, the noise may obey a random Gaussian normal distribution, and in the noise sampling subnetwork, current noise sampling data is obtained by sampling from the random Gaussian normal distribution.

[0123] In step S223, based on the current sampling time step, the current first weight corresponding to the original sample feature data and the current second weight corresponding to the current noise sampling data are determined from the preset noise scheduling table corresponding to the reference noise adding network.

[0124] Among them, the hyperparameters in the preset noise schedule are used to indicate the noise increase intensity corresponding to different time steps.

[0125] In a specific embodiment, the preset noise scheduling table includes a first hyperparameter corresponding to different time steps and a second hyperparameter corresponding to different time steps, wherein the sum of the first hyperparameter and the second hyperparameter corresponding to the same time step is 1, and it is assumed that the second hyperparameter can grow linearly with time, and it is assumed that the value of the second hyperparameter can increase from 0.0001 to 0.02; the third hyperparameter corresponding to the time step t can be the first hyperparameter corresponding to the time step 1, the first hyperparameter corresponding to the time step 2... The product value of the first hyperparameter corresponding to the time step t.

[0126] In a specific embodiment, the current first weight corresponding to the original sample feature data may be the arithmetic square root of the third hyperparameter corresponding to the time step t, and the calculation process of the current second weight corresponding to the current noise sampling data may include: first calculating the opposite of the third hyperparameter corresponding to the time step t, adding the opposite to 1 to obtain the sum of the two, and then calculating the arithmetic square root of the sum of the two, and using the calculated arithmetic square root as the current second weight corresponding to the current noise sampling data.

[0127] In step S224, the original sample feature data, the current first weight, the current noise sampling data and the current second weight are input into the noise addition subnetwork to perform noise addition processing to obtain the first sample feature data.

[0128] In a specific embodiment, the noise addition subnetwork is used to perform weighted summation processing, multiplying the original sample feature data with the current first weight corresponding to the original sample feature data to obtain a first value, multiplying the current noise sampling data with the current second weight corresponding to the current noise sampling data to obtain a second value, and then adding the first value and the second value to obtain the first sample feature data.

[0129] In step S225, a first sample noise image is obtained according to the first sample characteristic data.

[0130] It is feasible to map the first sample characteristic feature data to an image pixel space to obtain a first sample noise image in a pixel image format.

[0131] In a specific embodiment, when the first sample feature data is subsequently directly subjected to denoising or image generation processing, step S225 may be omitted.

[0132] In the above embodiment, the denoising process is controlled by using a preset noise scheduling table, so that only one random noise sampling and increase is required to achieve the effect of t-step denoising processing on the sample image, which can effectively improve the efficiency of image denoising and thus improve the training efficiency of the model.

[0133] In step S230, based on the reference denoising network and a preset number of unit time intervals, the first sample noise image is denoised to obtain a second sample noise image; the preset number is not greater than the number of unit time intervals in at least one unit time interval.

[0134] In the disclosed embodiment, the reference denoising network is mainly used to perform distillation training on the consistency model to be trained, so as to quickly learn the image generation ability of the reference denoising network. The reference denoising network uses a method of first predicting noise and then removing noise to perform multi-step denoising on the noisy image to generate an image.

[0135] In the disclosed embodiment, the preset number k needs to satisfy 1≤k≤t, where t is the current sampling time step, that is, the number of unit time intervals included in the current sampling time step is t. Based on the reference denoising network and the preset number of unit time intervals, the first sample noise image is denoised, that is, the first sample noise image is denoised for a preset number of steps through the reference denoising network to obtain the second sample noise image corresponding to the time step tk. Based on self-consistency, the first sample noise image and the second sample noise image are the noise addition results corresponding to different time steps on the same noise addition trajectory, and the first sample noise image and the second sample noise image can both be mapped to the starting point of the noise addition trajectory (that is, the sample image).

[0136] In another embodiment of the present disclosure, a time step t may be randomly sampled, and t+1 steps of forward denoising and one step of reverse denoising may be performed.

[0137] In one embodiment of the present disclosure, k is greater than 1, that is, a skipping-step strategy is adopted, which can avoid the output difference between the first sample noise image and the second sample noise image of the subsequent consistency model to be trained being too small, thereby affecting the speed of distillation training.

[0138] In another embodiment of the present disclosure, the number of steps k in the denoising process may also be dynamically sampled, and the sampling value satisfies 1≤k≤t.

[0139] In one embodiment of the present disclosure, when k is 1, the reference denoising network is constructed based on an ordinary differential solver, and the ordinary differential solver can be used to perform a one-step denoising solution to obtain a second sample noise image.

[0140] In one embodiment of the present disclosure, specifically, the reference denoising network includes a second feature representation subnetwork, a noise prediction subnetwork and a noise removal subnetwork, such as Figure 4 As shown, step S230 may include:

[0141] In step S231, the first sample noise image is input into the second feature representation subnetwork, and feature representation processing is performed to obtain first sample feature data; the first sample feature data is image feature data or latent space feature data.

[0142] In a specific embodiment, the first sample noise image is input into the second feature representation subnetwork, and the first sample feature data can be obtained through specific image feature extraction and embedding representation. When the reference image generation model generates images in the image pixel space, the first sample feature data is the image feature data; when the reference image generation model generates images in the low-dimensional latent space, the first sample feature data is the latent space feature data. It can be understood that image generation in the low-dimensional latent space can reduce resource consumption and accelerate model training.

[0143] In step S232, the first sample feature data and a preset number of unit time intervals are input into the noise prediction subnetwork to perform noise prediction processing to obtain current noise prediction data corresponding to the preset number of unit time intervals.

[0144] The preset number is not greater than the number of unit time intervals in at least one unit time interval.

[0145] In a specific embodiment, the noise prediction subnetwork is a neural network used to predict the noise data added in a preset number of unit time intervals within the current sampling time step. The value of the predicted noise data is related to the parameters of the noise prediction subnetwork, and is also related to the current sampling time step, the preset number and the first sample noise image.

[0146] In step S233, the first sample feature data and the current noise prediction data are input into the noise removal subnetwork to perform noise removal processing to obtain second sample feature data.

[0147] It is feasible to obtain the above-mentioned second sample characteristic data by subtracting the first sample characteristic data from the current noise preset data.

[0148] In step S234, a second sample noise image is obtained according to the second sample feature data.

[0149] Feasibly, the second sample characteristic feature data may be mapped to an image pixel space to obtain a second sample noise image in a pixel image format.

[0150] It should be noted that the second sample noise image is the result of k-step reverse denoising based on the first sample noise image. In essence, it is also based on partial generation of the sample image, so that the image generation ability of the reference image generation model can be learned.

[0151] In the above embodiment, a denoising process of a limited number of steps is performed using a reference denoising network to obtain a second sample noise image on the same noisy trajectory as the first sample noise image, so that subsequent consistency learning can be performed to learn the image generation capability of the reference image generation model. When the preset number is greater than 1, that is, the skipping-step strategy is adopted, it is possible to avoid the output difference of the subsequent consistency model to be trained based on the first sample noise image and the second sample noise image being too small, thereby improving the speed of distillation training.

[0152] In step S240, the first sample noise image is input into the consistency model to be trained, and image generation processing is performed to obtain a first sample prediction image.

[0153] In the disclosed embodiment, the consistency model to be trained can be a deep neural network model in any form, and the consistency model to be trained should satisfy the constraints of the consistency function after training, that is, the target image generation model obtained after training is a consistency model with self-consistency characteristics. The self-consistency characteristics satisfied by the consistency model to be trained after training can be expressed as follows: the noise image corresponding to any time step on the same noisy trajectory can be mapped to the same sample image through one-step denoising.

[0154] In the disclosed embodiment, the first sample noise image is input into the consistency model to be trained, and a direct image generation process is performed to obtain a first sample prediction image, which is the prediction result of the consistency model to be trained on the sample image based on the first sample noise image. That is, the consistency model to be trained can directly map noise data to image data to generate an image.

[0155] In one embodiment of the present disclosure, the consistency model to be trained may be a pre-trained Wensheng graph model, and the distillation training of the consistency model to be trained is the fine-tuning training of the pre-trained Wensheng graph model.

[0156] In one embodiment of the present disclosure, the consistency model to be trained includes a target noisy network and an image generation network to be trained. The target noisy network does not contain parameters that need to be trained, and the image generation network to be trained can be a deep neural network in any form.

[0157] In step S250, the second sample noise image is input into the consistency model to be trained, and image generation processing is performed to obtain a second sample prediction image.

[0158] In the disclosed embodiment, the second sample noise image is input into the consistency model to be trained, and a direct image generation process is performed to obtain a second sample prediction image, which is the prediction result of the consistency model to be trained on the sample image based on the second sample noise image. That is, the consistency model to be trained can directly map noise data to image data to generate an image.

[0159] In step S260, the dynamic weight corresponding to the current sampling time step is determined; the dynamic weight is negatively correlated with the current sampling time step.

[0160] In the embodiment of the present disclosure, if the current sampling time step is different, the dynamic weight will also be different in the current training round based on the sample image, and the dynamic weight is negatively correlated with the current sampling time step, that is, the dynamic weight decreases as the current sampling time step increases.

[0161] In the design and testing of the embodiments of the present disclosure, it was found that when the current sampling time step increases, the noise intensity increases overall, while the dynamic weight decreases. This can strengthen the consistency learning when the image has low noise (corresponding to a smaller sampling time step) and weaken the consistency learning when the image has high noise (corresponding to a larger sampling time step). Strengthening the consistency learning of low-noise images can strengthen the attention of the consistency model to be trained to image details and textures, thereby improving the quality of images generated by the model.

[0162] In one embodiment of the present disclosure, the dynamic weight can be set to any decreasing function with the time step as the independent variable. After the current sampling time step is determined, the dynamic weight can be determined, and the two are negatively correlated.

[0163] In another embodiment of the present disclosure, Figure 5 As shown, step S260 may include:

[0164] In step S261, a preset noise schedule corresponding to the reference noise adding network is obtained.

[0165] Among them, the hyperparameters in the preset noise schedule indicate the noise increase intensity corresponding to different time steps.

[0166] In a specific embodiment, referring to the aforementioned embodiment, considering that the first hyperparameter in the preset noise schedule in the reference noise adding network decreases with the increase of the time step, the first hyperparameter in the preset noise schedule can be directly used to design the dynamic weight.

[0167] In step S262, at least one current hyperparameter corresponding to at least one unit time interval is determined from a preset noise schedule.

[0168] Among them, the at least one current hyperparameter includes a first hyperparameter corresponding to a time step of 1, a first hyperparameter corresponding to a time step of 2, and a first hyperparameter corresponding to a time step of t.

[0169] In step S263, at least one current hyperparameter is accumulated and multiplied to obtain a dynamic weight corresponding to the current sampling time step.

[0170] That is, when the time step is the current sampling time step t, at least one current hyperparameter is continuously multiplied to obtain a dynamic weight.

[0171] In the above embodiment, considering that the first hyperparameter in the preset noise scheduling table in the reference noise adding network decreases with the increase of the time step, the first hyperparameter in the preset noise scheduling table can be directly used to design the dynamic weight. On the basis of step S223, the third hyperparameter calculated in advance can also be directly used as the dynamic weight corresponding to the current sampling time step, which can effectively improve the processing efficiency.

[0172] In step S270, target loss data is determined according to the first sample prediction image, the second sample prediction image and the dynamic weight.

[0173] In one embodiment of the present disclosure, step S270 may be implemented as follows:

[0174] In step S271, original loss data is determined according to the first sample prediction image and the second sample prediction image.

[0175] The original loss data may indicate the degree of image difference between the first sample prediction image and the second sample prediction image when the input is a sample image and the time step is the current sampling time step. Specifically, it may be calculated by a loss function such as a cross entropy loss function, a mean square error loss function, or a distance function.

[0176] In step S272, the original loss data is adjusted based on the dynamic weight to obtain the target loss data.

[0177] In a feasible implementation, the original loss data is numerically adjusted using the dynamic weight corresponding to the current sampling time step, and the target loss data between the first sample prediction image and the second sample prediction image can be obtained when the input is the sample image and the time step is the current sampling time step. Specifically, the dynamic weight corresponding to the current sampling time step and the above original loss data can be multiplied to obtain the target loss data between the first sample prediction image and the second sample prediction image when the input is the sample image and the time step is the current sampling time step.

[0178] In another embodiment of the present disclosure, in the case where there are multiple current sampling time steps, the original loss data may include first loss data corresponding to each current sampling time step, wherein the first loss data corresponding to each current sampling time step can be determined based on the first sample prediction image corresponding to each current sampling time step and the second sample prediction image corresponding to each current sampling time step, and the original loss data is obtained based on the first loss data corresponding to each current sampling time step; based on the dynamic weights corresponding to each current sampling time step, the first loss data corresponding to each current sampling time step in the original loss data is adjusted, specifically, the dynamic weights corresponding to each current sampling time step can be multiplied by the first loss data corresponding to each current sampling time step to obtain the second loss data corresponding to each current sampling time step, and the expected value of the second loss data corresponding to each current sampling time step is calculated to obtain the target loss data when the input is a sample image and the time step includes multiple current sampling time steps, and the target loss data can indicate the average processing performance of the consistency model to be trained when the input is a sample image.

[0179] In the above embodiment, the original loss data between the first sample prediction image and the second sample prediction image is first calculated, and the weighted target loss data is obtained by combining the dynamic weight corresponding to the current sampling time step, so that when the target loss data is used to train the consistency model to be trained, the attention of the consistency model to be trained to the details and textures of the image when the current sampling time step is relatively small can be strengthened, thereby improving the image generation quality of the target image generation model.

[0180] It should be noted that when training the image generation model in the latent space, the noise image or the predicted image can be represented as the corresponding latent space object. For example, the first sample noise image can be represented as z t , the second sample noise image can be expressed as I will not go into details about the rest.

[0181] In step S280, the consistency model to be trained is trained according to the target loss data to obtain a target image generation model.

[0182] In the disclosed embodiment, the consistency model to be trained is trained according to the target loss data, and the obtained target image generation model has the ability to directly generate images within a single unit time interval (i.e., a single step), which greatly reduces the number of steps for image generation. In the application stage of image generation, it can greatly save reasoning resources and reduce performance requirements while maintaining or improving the quality of image generation.

[0183] In a specific embodiment of the present disclosure, the gradient of the target loss data is calculated, and the parameters of the consistency model to be trained are adjusted along the direction of gradient descent until the target image generation model is obtained when the preset model convergence condition is met.

[0184] In one embodiment of the present disclosure, according to application requirements, the consistency model to be trained needs to be trained to generate a pixel image corresponding to a noise image under the constraints of a certain text. Accordingly, the training process can be implemented as follows:

[0185] In step S310: obtain prompt text.

[0186] In step S320: the first sample noise image and the prompt text are input into the consistency model to be trained, and image generation processing is performed with the prompt text as a constraint condition to obtain a first sample predicted image.

[0187] Specifically, the text feature extraction module embedded in the consistency model to be trained is used to perform feature extraction and embedding representation on the prompt text to obtain text feature data corresponding to the prompt text, and then, using the text feature data as a constraint condition, image generation processing is performed based on the first sample feature data corresponding to the first sample noise image to obtain a first sample predicted image.

[0188] In step S330: the second sample noise image and the prompt text are input into the consistency model to be trained, and image generation processing is performed with the prompt text as a constraint condition to obtain a second sample predicted image.

[0189] Step S330 may refer to step S320 and will not be described in detail here.

[0190] Before step S330, based on the reference denoising network, a preset number of unit time intervals and the prompt text, the first sample noise image may be subjected to denoising processing with the prompt text as a constraint condition to obtain a second sample noise image.

[0191] Based on the first sample prediction image, the sample noise image and the dynamic weight corresponding to the current sampling time step, the target loss data is determined. The specific process can refer to step S270; the target loss data is used to train the consistency model to be trained to obtain the target image generation model. In the above embodiment, the consistency model to be trained can be trained under the constraint of the prompt text, so that a text-generated image model with the ability to directly generate images can be obtained, which expands the application scope of the target image generation model.

[0192] In another embodiment of the present disclosure, the consistency model to be trained can also be an image generation model without classifier guidance to achieve better image generation effect under certain conditions. Similarly, when the image generation model is trained in the latent space, the noise image or the predicted image can be represented as the corresponding latent space object.

[0193] In the disclosed embodiment, the trained target image generation model has the ability to generate pixel images directly from noise images in a single step. In practical applications, a limited number of rounds of noise addition and image generation processing can be set to improve the quality of the generated image.

[0194] Specifically, the target image generation model includes a target noise addition network and a target generation network, such as Figure 6 As shown, the method may further include:

[0195] In step S410, an image to be processed is obtained.

[0196] In step S420, the image to be processed is input into the target image generation model, and a preset number of image generation processes are performed in the target image generation model to obtain a target generated image corresponding to the image to be processed.

[0197] In any one of the image generation processes, the following steps are performed:

[0198] In step S421, based on the target denoising network and the current random time step, the current image to be processed is denoised to obtain a current noisy image.

[0199] For the specific noise adding process, please refer to step S220, which will not be described in detail here.

[0200] In step S422, the current noise image is input into the target generation network, and image generation processing is performed to obtain the current candidate generated image.

[0201] The target image generation model has the ability to directly generate images, that is, the target generation network therein can directly map the current noisy image which is noise data to the current candidate generated image of pixel data.

[0202] Among them, in the first image generation processing process, the current image to be processed is the image to be processed; in the non-first image generation processing process, the current image to be processed is the current candidate generated image obtained in the image generation processing round before the current image generation processing round.

[0203] Among them, by generating images multiple times and continuously iteratively updating the current candidate generated image, the quality of the final target generated image can be improved. Exemplarily, the preset number of times can be between 4 and 8, so as to achieve a balance between generation speed and generation quality.

[0204] In the above embodiment, during the inference process of the target image generation model, by alternately adding noise and generating images at different time steps, the quality of the generated image can be improved without affecting the generation efficiency.

[0205] It can be seen from the above embodiments that an embodiment of the present disclosure provides a training method for an image generation model. The present disclosure first performs a noise processing on the acquired sample image based on the reference denoising network in the reference image generation model and the current sampling time step corresponding to the reference denoising network to obtain a first sample noise image, wherein the current sampling time step includes at least one unit time interval; based on the reference denoising network in the reference image generation model and a preset number of unit time intervals, the first sample noise image is denoised to obtain a second sample noise image, and the preset number is not greater than the number of unit time intervals included in the current sampling time step; then the first sample noise image is input into the consistency model to be trained, image generation processing is performed to obtain a first sample prediction image, and the second sample noise image is input into the consistency model to be trained, image generation processing is performed to obtain a second sample prediction image; in the process of determining the loss data, a dynamic weight that is negatively correlated with the current sampling time step is introduced, and the target loss data determined according to the first sample prediction image, the second sample prediction image and the dynamic weight is used to train the consistency model to be trained, so as to obtain a target image generation model with the ability to directly generate images.

[0206] In the technical solution provided by the embodiment of the present disclosure, a reference image generation model is used to perform distillation training on a consistency model to be trained, so as to learn the image generation capability of the reference image generation model, and the amount of training can be reduced by sampling time steps, thereby improving the training efficiency. In the technical solution provided by the embodiment of the present disclosure, target loss data is calculated based on the self-consistency characteristics that the consistency model to be trained needs to satisfy, and the consistency model to be trained is trained using the target loss data. The obtained target image generation model can directly generate images within a single unit time interval (i.e., a single step), which greatly reduces the number of steps for image generation, and can greatly save inference resources and reduce performance requirements in the application stage of image generation. In the technical solution provided by the embodiment of the present disclosure, the target loss data introduces a dynamic weight that is negatively correlated with the current sampling time step, i.e., the dynamic weight decreases as the current sampling time step increases, which can enhance the attention of the consistency model to be trained to the details and texture of the image when the current sampling time step is relatively small, thereby improving the image generation quality of the target image generation model.

[0207] Figure 7 FIG. 1 is a block diagram of a training device for an image generation model according to an exemplary embodiment. Figure 7 , the device comprises:

[0208] An acquisition module 710 is configured to acquire a sample image, a reference image generation model, and a consistency model to be trained, wherein the reference image generation model includes a reference denoising network and a reference denoising network;

[0209] The denoising module 720 is configured to perform denoising processing on the sample image based on the reference denoising network and the current sampling time step corresponding to the reference denoising network to obtain a first sample noise image; the current sampling time step includes at least one unit time interval;

[0210] The denoising module 730 is configured to perform denoising processing on the first sample noise image based on the reference denoising network and a preset number of unit time intervals to obtain a second sample noise image; the preset number is not greater than the number of unit time intervals in the at least one unit time interval;

[0211] The first prediction module 740 is configured to input the first sample noise image into the consistency model to be trained, perform image generation processing, and obtain a first sample prediction image;

[0212] The second prediction module 750 is configured to input the second sample noise image into the consistency model to be trained, perform image generation processing, and obtain a second sample prediction image;

[0213] A weight determination module 760 is configured to determine a dynamic weight corresponding to the current sampling time step; the dynamic weight is negatively correlated with the current sampling time step;

[0214] A target loss calculation module 770 is configured to determine target loss data according to the first sample prediction image, the second sample prediction image and the dynamic weight;

[0215] The training module 780 is configured to execute training on the consistency model to be trained according to the target loss data to obtain a target image generation model.

[0216] In an optional embodiment, the weight determination module 760 includes:

[0217] A first acquisition unit is configured to execute acquisition of a preset noise schedule corresponding to the reference noise adding network; the hyperparameters in the preset noise schedule indicate the noise increase intensity corresponding to different time steps; the noise increase intensity is negatively correlated with the unit time interval;

[0218] A hyperparameter determination unit, configured to determine at least one current hyperparameter corresponding to the at least one unit time interval from the preset noise schedule;

[0219] The dynamic weight calculation unit is configured to perform cumulative multiplication of the at least one current hyperparameter to obtain the dynamic weight corresponding to the current sampling time step.

[0220] In an optional embodiment, the target loss calculation module 770 includes:

[0221] an original loss data determining unit, configured to determine original loss data according to the first sample prediction image and the second sample prediction image;

[0222] The target loss data determination unit is configured to adjust the original loss data based on the dynamic weight to obtain the target loss data.

[0223] In an optional embodiment, the reference noise adding network includes a first feature representation subnetwork, a noise sampling subnetwork and a noise adding subnetwork, and the noise adding module 720 includes:

[0224] A first feature representation unit is configured to input the sample image into the first feature representation subnetwork, perform feature representation processing, and obtain original sample feature data; the original sample feature data is image feature data or latent space feature data;

[0225] A noise sampling unit is configured to perform noise sampling processing based on the noise sampling subnetwork to obtain current noise sampling data;

[0226] A noise weight determination unit is configured to determine, based on the current sampling time step, a current first weight corresponding to the original sample feature data and a current second weight corresponding to the current noise sampling data from a preset noise scheduling table corresponding to the reference noise adding network; the hyperparameters in the preset noise scheduling table are used to indicate the noise increase intensity corresponding to different time steps;

[0227] A noise adding unit is configured to input the original sample feature data, the current first weight, the current noise sampling data and the current second weight into the noise adding sub-network, perform noise adding processing, and obtain the first sample feature data;

[0228] The first sample noise image determining unit is configured to obtain the first sample noise image according to the first sample feature data.

[0229] In an optional embodiment, the reference denoising network includes a second feature representation subnetwork, a noise prediction subnetwork and a noise removal subnetwork, and the denoising module 730 includes:

[0230] A second feature representation unit is configured to input the first sample noise image into the second feature representation subnetwork, perform feature representation processing, and obtain first sample feature data; the first sample feature data is image feature data or latent space feature data;

[0231] A noise prediction unit is configured to input the first sample feature data and the preset number of unit time intervals into the noise prediction subnetwork, perform noise prediction processing, and obtain current noise prediction data corresponding to the preset number of unit time intervals;

[0232] a noise removal unit configured to input the first sample feature data and the current noise prediction data into the noise removal subnetwork, perform noise removal processing, and obtain second sample feature data;

[0233] The second sample noise image determining unit is configured to obtain the second sample noise image according to the second sample feature data.

[0234] In an optional embodiment, the device further comprises:

[0235] A second acquisition unit is configured to execute acquisition of prompt text;

[0236] The first prediction module comprises:

[0237] A first condition generating unit is configured to input the first sample noise image and the prompt text into the consistency model to be trained, perform image generation processing with the prompt text as a constraint condition, and obtain the first sample prediction image;

[0238] The second prediction module comprises:

[0239] The first condition generating unit is configured to execute the image generation processing with the prompt text as a constraint condition on the second sample noise image and the prompt text, so as to obtain the second sample predicted image.

[0240] In an optional embodiment, the target image generation model includes a target noise addition network and a target generation network, and the device further includes:

[0241] A third acquisition unit is configured to acquire an image to be processed;

[0242] The image generation processing unit is configured to input the image to be processed into the target image generation model, perform a preset number of image generation processes in the target image generation model, and obtain a target generated image corresponding to the image to be processed; wherein, during any one of the image generation processes, the following steps are performed:

[0243] A random noise adding subunit is configured to perform noise adding processing on the current image to be processed based on the target noise adding network and the current random time step to obtain a current noisy image;

[0244] An image generation subunit is configured to input the current noise image into the target generation network, perform image generation processing, and obtain a current candidate generated image;

[0245] Among them, in the first image generation processing process, the current image to be processed is the image to be processed; in the non-first image generation processing process, the current image to be processed is the current candidate generation image obtained in the image generation processing round before the current image generation processing round.

[0246] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0247] Figure 8 is a block diagram of an electronic device for implementing a training method for an image generation model according to an exemplary embodiment. The electronic device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 8As shown. The electronic device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a training method for an image generation model is implemented. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a key, trackball or touchpad provided on the housing of the electronic device, or an external keyboard, touchpad or mouse, etc.

[0248] Fig. 9 is a block diagram of an electronic device for implementing a training method for an image generation model according to an exemplary embodiment. The electronic device may be a server, and its internal structure diagram may be as shown in FIG. Fig. 9 As shown. The electronic device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a training method for an image generation model is implemented.

[0249] Those skilled in the art will understand that Figure 8 and Fig. 9 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present disclosure, and does not constitute a limitation on the electronic device to which the scheme of the present disclosure is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0250] In an exemplary embodiment, an electronic device is also provided, including: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement a training method for an image generation model as in an embodiment of the present disclosure.

[0251] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the training method of the image generation model in the embodiment of the present disclosure.

[0252] In an exemplary embodiment, a computer program product is also provided, including computer instructions, which, when executed by a processor, implement the training method of the image generation model in the embodiment of the present disclosure.

[0253] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0254] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0255] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A training method for an image generation model, characterized in that: The method comprises: Acquire a sample image, a reference image generation model, and a consistency model to be trained, wherein the reference image generation model includes a reference denoising network and a reference denoising network; Based on the reference denoising network and the current sampling time step corresponding to the reference denoising network, the sample image is subjected to denoising processing to obtain a first sample noise image; the current sampling time step includes at least one unit time interval; Based on the reference denoising network and a preset number of unit time intervals, denoising the first sample noise image to obtain a second sample noise image; the preset number is not greater than the number of unit time intervals in the at least one unit time interval; Inputting the first sample noise image into the consistency model to be trained, performing image generation processing, and obtaining a first sample prediction image; Inputting the second sample noise image into the consistency model to be trained, performing image generation processing, and obtaining a second sample prediction image; Determine a dynamic weight corresponding to the current sampling time step; the dynamic weight is negatively correlated with the current sampling time step; Determining target loss data according to the first sample prediction image, the second sample prediction image and the dynamic weight; The consistency model to be trained is trained according to the target loss data to obtain a target image generation model.

2. The method according to claim 1, characterized in that The determining the dynamic weight corresponding to the current sampling time step includes: Obtaining a preset noise schedule corresponding to the reference noise adding network; the hyperparameters in the preset noise schedule indicate the noise increase intensity corresponding to different time steps; the noise increase intensity is negatively correlated with the unit time interval; Determining at least one current hyperparameter corresponding to the at least one unit time interval from the preset noise schedule; The at least one current hyperparameter is cumulatively multiplied to obtain the dynamic weight corresponding to the current sampling time step.

3. The method according to claim 1, characterized in that The determining target loss data according to the first sample prediction image, the second sample prediction image and the dynamic weight comprises: Determining original loss data according to the first sample prediction image and the second sample prediction image; Based on the dynamic weight, the original loss data is adjusted to obtain the target loss data.

4. The method according to claim 1, characterized in that: The reference noisy network includes a first feature representation subnetwork, a noise sampling subnetwork and a noise addition subnetwork, and the sample image is subjected to noise processing based on the reference noisy network and the current sampling time step corresponding to the reference noisy network to obtain a first sample noise image, including: Inputting the sample image into the first feature representation subnetwork, performing feature representation processing, and obtaining original sample feature data; the original sample feature data is image feature data or latent space feature data; Based on the noise sampling subnetwork, noise sampling processing is performed to obtain current noise sampling data; Based on the current sampling time step, determining the current first weight corresponding to the original sample feature data and the current second weight corresponding to the current noise sampling data from the preset noise scheduling table corresponding to the reference noise adding network; the hyperparameters in the preset noise scheduling table are used to indicate the noise increase intensity corresponding to different time steps; Input the original sample feature data, the current first weight, the current noise sampling data and the current second weight into the noise addition subnetwork to perform noise addition processing to obtain first sample feature data; The first sample noise image is obtained according to the first sample characteristic data.

5. The method according to claim 1, characterized in that: The reference denoising network includes a second feature representation subnetwork, a noise prediction subnetwork and a noise removal subnetwork. The denoising process is performed on the first sample noise image based on the reference denoising network and a preset number of unit time intervals to obtain a second sample noise image, including: Inputting the first sample noise image into the second feature representation subnetwork, performing feature representation processing, and obtaining first sample feature data; the first sample feature data is image feature data or latent space feature data; Inputting the first sample feature data and the preset number of unit time intervals into the noise prediction subnetwork, performing noise prediction processing, and obtaining current noise prediction data corresponding to the preset number of unit time intervals; Inputting the first sample feature data and the current noise prediction data into the noise removal subnetwork to perform noise removal processing to obtain second sample feature data; The second sample noise image is obtained according to the second sample characteristic data.

6. The method according to claim 1, characterized in that The method further comprises: Get the prompt text; The step of inputting the first sample noise image into the consistency model to be trained and performing image generation processing to obtain a first sample prediction image includes: Inputting the first sample noise image and the prompt text into the consistency model to be trained, performing image generation processing with the prompt text as a constraint condition, and obtaining the first sample prediction image; The step of inputting the second sample noise image into the consistency model to be trained and performing image generation processing to obtain a second sample prediction image includes: The second sample noise image and the prompt text are input into the consistency model to be trained, and image generation processing is performed with the prompt text as a constraint condition to obtain the second sample predicted image.

7. The method according to claim 1, characterized in that The target image generation model includes a target noise adding network and a target generation network, and the method further includes: Get the image to be processed; The image to be processed is input into the target image generation model, and image generation processing is performed a preset number of times in the target image generation model to obtain a target generated image corresponding to the image to be processed; wherein, during any one of the image generation processings, the following steps are performed: Based on the target noise adding network and the current random time step, performing noise adding processing on the current image to be processed to obtain a current noisy image; Inputting the current noise image into the target generation network to perform image generation processing to obtain a current candidate generated image; Among them, in the first image generation processing process, the current image to be processed is the image to be processed; in the non-first image generation processing process, the current image to be processed is the current candidate generation image obtained in the image generation processing round before the current image generation processing round.

8. A training device for an image generation model, characterized in that: The device comprises: An acquisition module is configured to acquire a sample image, a reference image generation model, and a consistency model to be trained, wherein the reference image generation model includes a reference denoising network and a reference denoising network; A denoising module is configured to perform denoising processing on the sample image based on the reference denoising network and a current sampling time step corresponding to the reference denoising network to obtain a first sample noise image; the current sampling time step includes at least one unit time interval; a denoising module configured to perform denoising processing on the first sample noise image based on the reference denoising network and a preset number of unit time intervals to obtain a second sample noise image; the preset number is not greater than the number of unit time intervals in the at least one unit time interval; A first prediction module is configured to input the first sample noise image into the consistency model to be trained, perform image generation processing, and obtain a first sample prediction image; A second prediction module is configured to input the second sample noise image into the consistency model to be trained, perform image generation processing, and obtain a second sample prediction image; A weight determination module is configured to determine a dynamic weight corresponding to the current sampling time step; the dynamic weight is negatively correlated with the current sampling time step; a target loss calculation module, configured to determine target loss data according to the first sample prediction image, the second sample prediction image and the dynamic weight; The training module is configured to train the consistency model to be trained according to the target loss data to obtain a target image generation model.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the training method of the image generation model as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the training method of the image generation model as described in any one of claims 1 to 7.

11. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the training method of the image generation model described in any one of claims 1 to 7 is implemented.

Images