Sampling model training method, image processing method and related device

Abstract

The embodiment of the invention discloses a sampling model training method, an image processing method and a related device, and the method comprises the steps: determining an initial image and first starting time; performing noise addition on the initial image based on the first starting time to obtain a first training image; performing sampling prediction based on the first training image through a sampling model to obtain a first sampling step length; denoising the first training image based on the first sampling step length through a diffusion model to obtain a first sample image at the first sample time; performing noise addition on the initial image based on the first sample time to obtain a first expected image; and training the sampling model according to a loss function term, wherein the loss function term indicates the similarity between the first sample image and the first expected image. The trained sampling model can be applied to a sampling scene of a diffusion model, the sampling step length of the image in each intermediate state is dynamically allocated, and the problem of low sampling efficiency of the diffusion model in an image processing process is solved.

Description

A method for training a sampling model, an image processing method, and related apparatus. Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method for training a sampling model, an image processing method, and related apparatus. Background Technology

[0002] In recent years, diffusion models (DM), as a new and noteworthy branch of visual generative models, have gradually surpassed the historical importance of generative adversarial networks (GANs) due to their unique advantage of producing high-quality and highly diverse generated results. DM has achieved groundbreaking progress in several cutting-edge fields, including text-to-image, text-to-video, and text-to-3D model generation, demonstrating its powerful cross-modal generation capabilities. It has also shown remarkable generative abilities in image super-resolution. Specifically, in tasks such as face super-resolution and general image super-resolution, diffusion models excel at handling complex images in real-world scenes. This is thanks to the rich prior knowledge learned on large-scale datasets, which greatly helps the model achieve superior performance in detail recovery of high-resolution images.

[0003] However, a significant characteristic of diffusion models is that they follow a Markov process, meaning that during generation, the model needs to iterate through denoising steps to gradually approximate the final generated result. Typically, this process requires thousands (e.g., 1000 or more) iterations to ensure the quality and diversity of the generated images. This complex computational load undoubtedly poses a major challenge to the practical application of diffusion models, severely limiting their widespread adoption in large-scale datasets and real-time applications. Therefore, to overcome this challenge, current research urgently needs an efficient sampling method applied to diffusion models to significantly reduce computation time and resource consumption while maintaining generation quality. Summary of the Invention

[0004] This application provides a method for training a sampling model, an image processing method, and related apparatus to solve the problem of low sampling efficiency of diffusion models in image processing.

[0005] The first aspect of this application provides a method for training a sampling model, comprising:

[0006] Determine the initial image and the first start time;

[0007] The initial image is noise-added based on the first start time to obtain the first training image;

[0008] The first sampling step size is obtained by sampling prediction based on the first training image using the sampling model;

[0009] The first training image is denoised using a diffusion model based on a first sampling step size to obtain the first sample image at the first sample time.

[0010] Noise is added to the initial image based on the first sample time to obtain the first desired image;

[0011] The sampling model is trained based on a loss function term, which indicates the similarity between the first sample image and the first desired image.

[0012] The method provided in this application embodiment is used to train a sampling model. The trained sampling model can be applied to the sampling scenario of the diffusion model, so that the sampling model can predict the image at any intermediate state during the image diffusion process, dynamically allocate the next sampling step size, and use the sampling model as the prediction guide for the diffusion model sampler. Different sampling time series can be matched for each different input, so as to ensure the quality of diffusion model generation, while greatly reducing the computation time and resource consumption, and improving the generation efficiency of diffusion model.

[0013] In one possible implementation, a first sampling step size is obtained by performing sampling prediction based on the first training image using a sampling model, including:

[0014] The sampling model performs sampling prediction based on the first training image to obtain multiple first sampling step sizes, which correspond to the first sample image and multiple first expected image at multiple first sample times, respectively.

[0015] After adding noise to the initial image based on the first sample time to obtain the first desired image, the process also includes:

[0016] Calculate the similarity between the corresponding multiple first sample images and multiple first expected images;

[0017] The first sample time corresponding to the highest similarity is determined as the second starting time, and noise is added to the initial image based on the second starting time to obtain the second training image. The second training image corresponds to the second sample image and the second expected image under the second sampling step and the second sample time.

[0018] The loss function term specifically indicates the similarity between the corresponding sample image and the expected image. The sample image includes the first sample image and the second sample image, and the expected image includes the first expected image and the second expected image.

[0019] In one possible implementation, the second training image specifically corresponds to multiple second sampling steps, and the multiple second sampling steps respectively correspond to the second sample image and multiple second expected images at multiple second sample times.

[0020] In one possible implementation, the first sample time corresponding to the highest similarity is determined as the second starting time, and noise is added to the initial image based on the second starting time, specifically including:

[0021] The time of the first sample corresponding to the highest similarity is determined as the second starting time;

[0022] Noise is added to the initial image based on the second start time until the number of predictions by the sampling model reaches a preset value.

[0023] In one possible implementation, before determining the first sample time corresponding to the highest similarity as the second starting time, and before adding noise to the initial image based on the second starting time, the method further includes:

[0024] The total step size is divided into multiple time periods, and the total step size is the step size for spreading the initial image to a completely noisy image.

[0025] The first sample time corresponding to the highest similarity is determined as the second starting time, and noise is added to the initial image based on the second starting time, specifically including:

[0026] Determine the target time period to which the first start time belongs; the target time period may belong to multiple time periods.

[0027] If the second start time is not less than the minimum time of the target time period, then noise is added to the initial image based on the second start time.

[0028] In one possible implementation, a first sampling step size is obtained by performing sampling prediction based on the first training image using a sampling model, including:

[0029] The sampling model performs sampling prediction based on the first training image to obtain multiple first sampling step sizes. The multiple first sampling step sizes include a first sub-sampling step size and a second sub-sampling step size, and the first sub-sampling step size is greater than the second sub-sampling step size.

[0030] Before training the sampling model based on the loss function term, the following is also included:

[0031] Calculate the first similarity between the first subsample image and the first sub-expected image. The first subsample image is obtained by denoising the first training image based on the first subsample step size. The first sub-expected image is obtained by adding noise to the initial image based on the first subsample time corresponding to the first subsample image.

[0032] Calculate the second similarity between the second subsample image and the second subtraining image. The second subsample image is obtained by denoising the second training image based on the second subsampling step size. The second subexpected image is obtained by adding noise to the initial image based on the second subsample time corresponding to the second subsample image.

[0033] If the difference between the first similarity and the second similarity is less than a preset value, the first similarity is increased based on the preset reward function.

[0034] In one possible implementation, the reward function indicates a first weight and a second weight, the first weight corresponding to a first subsampling step size and the second weight corresponding to a second subsampling step size;

[0035] The first weight is greater than the second weight.

[0036] A second aspect of this application provides an image processing method, comprising:

[0037] Acquire the first noisy image;

[0038] The sampling model predicts the first sampling step size based on the first noisy image, and the sampling model is trained based on the training method of the sampling model of any one of claims 1 to 7;

[0039] The first noisy image is denoised using a diffusion model based on a first sampling step size to obtain a second noisy image.

[0040] In one possible implementation, after denoising the first noisy image using a diffusion model based on a first sampling step size to obtain the first image, the method further includes:

[0041] The second sampling step size is predicted based on the second noisy image using a sampling model.

[0042] The third image is obtained by denoising the second noisy image using a diffusion model based on the second sampling step size.

[0043] A third aspect of this application provides a training apparatus for a sampling model, comprising:

[0044] The determination module is used to determine the initial image and the first start time;

[0045] The noise-adding module is used to add noise to the initial image based on the initial time to obtain the first training image;

[0046] The prediction module is used to perform sampling prediction based on the first training image using the sampling model to obtain the first sampling step size;

[0047] The denoising module is used to denoise the first training image based on the first sampling step size using a diffusion model to obtain the first sample image at the first sample time.

[0048] The noise-adding module is also used to add noise to the initial image based on the first sample time to obtain the first desired image;

[0049] The training module is used to train the sampling model based on a loss function term, which indicates the similarity between the first sample image and the first desired image.

[0050] In one possible implementation, the prediction module is specifically used to perform sampling prediction based on the first training image through the sampling model to obtain multiple first sampling step sizes, which correspond to the first sample image and multiple first expected image at multiple first sample times, respectively.

[0051] It also includes: a calculation module, used to calculate the similarity between the corresponding multiple first sample images and multiple first expected images; to determine the first sample time corresponding to the highest similarity as the second start time, and to add noise to the initial image based on the second start time to obtain the second training image, wherein the second training image corresponds to the second sample image and the second expected image under the second sampling step and the second sample time;

[0052] The loss function term specifically indicates the similarity between the corresponding sample image and the expected image. The sample image includes the first sample image and the second sample image, and the expected image includes the first expected image and the second expected image.

[0053] In one possible implementation, the second training image specifically corresponds to multiple second sampling steps, and the multiple second sampling steps respectively correspond to the second sample image and multiple second expected images at multiple second sample times.

[0054] In one possible implementation, the calculation module is specifically used to determine the first sample time corresponding to the highest similarity as the second starting time; and to add noise to the initial image based on the second starting time until the number of predictions of the sampling model reaches a preset value.

[0055] In another possible implementation,

[0056] The determination module is also used to divide the total step size into multiple time periods, where the total step size is the step size for spreading the initial image to a completely noisy image;

[0057] The denoising module is specifically used to determine the target time period to which the first start time belongs, and the target time period belongs to multiple time periods; if the second start time is not less than the minimum time of the target time period, then noise is added to the initial image based on the second start time.

[0058] In one possible implementation, the prediction module is specifically used to perform sampling prediction based on the first training image through a sampling model to obtain multiple first sampling step sizes, the multiple first sampling step sizes including a first sub-sampling step size and a second sub-sampling step size, the first sub-sampling step size being larger than the second sub-sampling step size.

[0059] The calculation module is specifically used to calculate the first similarity between the first subsample image and the first sub-expected image. The first subsample image is obtained by denoising the first training image based on the first subsampling step size, and the first sub-expected image is obtained by adding noise to the initial image based on the first subsample time corresponding to the first subsample image. The module also calculates the second similarity between the second subsample image and the second subtraining image. The second subsample image is obtained by denoising the second training image based on the second subsampling step size, and the second sub-expected image is obtained by adding noise to the initial image based on the second subsample time corresponding to the second subsample image. If the difference between the first similarity and the second similarity is less than a preset value, the first similarity is increased based on a preset reward function.

[0060] In one possible implementation, the reward function indicates a first weight and a second weight, the first weight corresponding to a first subsampling step size and the second weight corresponding to a second subsampling step size; the first weight is greater than the second weight.

[0061] A fourth aspect of this application provides an image processing apparatus, comprising:

[0062] The acquisition module is used to acquire the first noisy image;

[0063] A sampling module is used to predict a first sampling step size based on the first noisy image using a sampling model, wherein the sampling model is trained based on the training method of the sampling model described in the first aspect above.

[0064] A denoising module is used to denoise the first noisy image based on the first sampling step size.

[0065] In one possible implementation,

[0066] The sampling module is also used to predict the second sampling step size based on the second noisy image using the sampling model;

[0067] The denoising module is also used to denoise the second noisy image based on the second sampling step size using a diffusion model to obtain the third image.

[0068] A fifth aspect of this application provides a computing device, including a processor;

[0069] The processor is used to execute computer programs or computer instructions in memory to perform any of the methods described above.

[0070] In one possible implementation, the memory is also included.

[0071] The sixth aspect of this application provides a computer program product containing instructions that, when executed by a computing device, cause the computing device to perform the methods of any of the above aspects.

[0072] A seventh aspect of this application provides a computer-readable storage medium, characterized in that it includes computer program instructions, which, when executed by a computing device, perform the method of any of the above aspects.

[0073] The beneficial effects of the technical solutions provided in aspects two through seven above can be referred to the beneficial effects of the technical solutions in aspect one, and will not be repeated here. Attached Figure Description

[0074] Figure 1 is a flowchart of the training method for the sampling model provided in the embodiment of this application;

[0075] Figure 2 is a flowchart of the training method for the sampling model provided in the embodiment of this application;

[0076] Figure 3 is a flowchart of the image processing method provided in an embodiment of this application;

[0077] Figure 4 is a schematic diagram of the application of the sampling model obtained by the training method of the sampling model provided in the embodiment of this application;

[0078] Figure 5 is a schematic diagram comparing the performance of the sampling method optimized by the sampling model provided in this application on two datasets;

[0079] Figure 6 is a schematic diagram comparing the performance of multiple different sampling strategies;

[0080] Figure 7 is a schematic diagram of the structure of the training device for the sampling model provided in the embodiment of this application;

[0081] Figure 8 is a schematic diagram of the image processing apparatus provided in an embodiment of this application;

[0082] Figure 9 is a schematic diagram of the structure of the computing device provided in the embodiment of this application. Detailed Implementation

[0083] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application are described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Those skilled in the art will understand that with the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0084] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices. The naming or numbering of steps appearing in this application does not imply that the steps in the method flow must be performed in the chronological / logical order indicated by the naming or numbering. The execution order of named or numbered process steps can be changed according to the desired technical purpose, as long as the same or similar technical effect is achieved. The division of units in this application is a logical division. In practical applications, there may be other division methods. For example, multiple units may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the shown or discussed mutual coupling, direct coupling, or communication connection may be through some interface, and the indirect coupling or communication connection between units may be electrical or other similar forms, none of which are limited in this application. Furthermore, the units or sub-units described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed among multiple circuit units. Some or all of the units can be selected to achieve the purpose of the solution in this application according to actual needs.

[0085] Diffusion models are a type of generative model that progressively adds noise to data and then learns how to gradually denoise the noisy data to reconstruct the original data, thus generating new data samples. In recent years, diffusion models (DM), as a new and noteworthy branch of visual generative models, have gradually surpassed the historical importance of generative adversarial networks (GANs) due to their unique advantage of producing high-quality and highly diverse generated results. DM has not only achieved breakthroughs in several cutting-edge fields such as text-to-image (text-to-image), text-to-video (text-to-video), and text-to-3D model (text-to-3D), demonstrating its powerful cross-modal generative capabilities, but has also shown remarkable generative capabilities in the field of image super-resolution. Specifically, in specific tasks such as face super-resolution and general image super-resolution, diffusion models are particularly adept at handling complex images facing real-world scenes. This is thanks to the rich prior knowledge learned on large-scale datasets, which greatly helps the model achieve excellent performance in detail recovery of high-resolution images.

[0086] However, a significant characteristic of diffusion models is that they follow a Markov process, meaning that during generation, the model needs to iterate through denoising steps to gradually approximate the final generated result. Typically, this process requires thousands (e.g., 1000 or more) iterations to ensure the quality and diversity of the generated images. This complex computational load undoubtedly poses a major challenge to the practical application of diffusion models, severely limiting their widespread adoption in large-scale datasets and real-time applications. Therefore, to overcome this challenge, current research urgently needs an efficient sampling method applied to diffusion models to significantly reduce computation time and resource consumption while maintaining generation quality.

[0087] To address the above issues, the following solutions have been proposed:

[0088] Denoising diffusion implicit models (DIMM): This method defines a non-Markovian sampling process, which can be viewed as solving a first-order ordinary differential equation (ODE). This allows for the sampling process from noise to data in fewer steps, meaning it can sample a shorter subsequence than traditional DDPM. However, when the number of sampling steps is reduced, the performance of DDIM may decrease significantly because fewer steps may not be sufficient to completely remove noise or generate sufficiently clear data.

[0089] Autodiffusion is an innovative method for accelerating automated diffusion models by optimizing time steps and network architectures without requiring training. It significantly improves sampling efficiency by optimizing time steps and network architectures without the need for training. First, Autodiffusion defines a broad search space encompassing both time step and network structure searches. Then, it employs a genetic algorithm to continuously generate new individuals through crossover and mutation. For each individual, the method evaluates its performance using the Frechet Inception Distance (FID); a lower FID value indicates better performance. The top 50 individuals are retained and further crossover and mutation are performed iteratively to find the optimal time step. Compared to traditional default sampling methods (such as uniform sampling sequences), Autodiffusion's time series searched using a genetic algorithm are more reasonable and efficient. This is because uniform sampling sequences may not always be optimal, especially on different models and datasets. By using genetic algorithms, the Autodiffusion method can find sampling sequences that are more suitable for the current model and data, thus effectively improving sampling results. However, when evaluating the quality of individuals, this method requires sampling a large number of samples to calculate the FID, resulting in an exceptionally time-consuming search process.

[0090] Adaptive Step Selection for Fast Diffusion (AdaDiff): This method trains a small classification model that allows for different sampling step strategies for input prompts of varying difficulty, thus achieving higher average sampling efficiency. Its advantage lies in its ability to automatically select different sampling steps for input prompts of varying difficulty, offering greater flexibility. However, this classification model can only select a fixed number of sampling steps based on the difficulty of the input samples, resulting in a fixed, uniform sampling sequence, which fails to achieve optimal performance.

[0091] In summary, although diffusion models demonstrate powerful generative capabilities, their complex computational requirements severely limit their application.

[0092] To address the aforementioned issues, this application provides a sampling model training method, an image processing method, and related apparatus. This sampling model can predict the sampling step size of an image in various states during the image denoising process using a diffusion model, enabling the diffusion model to dynamically select the sampling step size, thereby effectively improving the sampling effect and efficiency of the diffusion model.

[0093] The training method of the sampling model provided in the embodiments of this application will be introduced next. This sampling model is used for the selection of the sampling step size in the diffusion model.

[0094] Please refer to Figure 1, which is a flowchart of the training method for the sampling model provided in the embodiments of this application, including:

[0095] 101. Determine the initial image and the first start time.

[0096] The initial image refers to the specified image used to train the model. It can be any image, such as a landscape photo, a portrait photo, a photo of an animal or plant photo, etc. The source of the initial image can be an open database created for training the model.

[0097] Understandably, in diffusion models, there is usually a concept of a time step, which represents the process from a completely clear image to a completely noisy image. This process is usually simulated as a continuous time series, where each time step adds a certain amount of noise to the image according to a certain probability distribution, corresponding to a change in the image state.

[0098] In this embodiment of the application, the first start time can be any time in this time series. The first start time can be manually specified or random.

[0099] 102. Add noise to the initial image based on the first start time to obtain the first training image.

[0100] Understandably, during the training of a diffusion model, the model needs to learn how to progressively reconstruct a sharp image from a completely noisy image. This typically involves learning a conditional probability distribution that describes how to predict the previous, sharper image state from the current noisy image state at different time steps.

[0101] In this embodiment, after determining the initial image and the first start time, adding noise to the initial image based on the first start time means adding noise to the initial image to the image state corresponding to the first start time. The resulting first training image should then be an image with a certain amount of noise. For example, if the process of transforming an image from a completely clear image to a completely noisy image requires 1000 time steps, and the first start time is 900, it can be understood that the first training image refers to the initial image being noisy to the degree of 90%.

[0102] 103. The first sampling step size is obtained by performing sampling prediction based on the first training image using the sampling model.

[0103] Understandably, during the training of the diffusion model, the training objective is to decompose the image generation process through a series of small "denoising" steps, thereby gradually generating high-quality samples. In the forward pass, noise is gradually added to the original image until it becomes a completely noisy image. This process is fixed and has no parameters to learn. In the backward pass, the model needs to learn how to gradually remove noise from the noisy image to restore a clear image. This process requires the model to learn a conditional probability distribution, i.e., how to predict a clearer image from the current noisy image at different time steps. During training, the model needs to continuously learn how to generate clear images from noisy samples. The sampling process provides the model with a continuous stream of noisy samples, which are used as training data to help the model learn the conditional probability distribution in the backward denoising process.

[0104] A sampling model can specifically be a classification model that predicts the next time step based on the current input features. This sampling model can be constructed by stacking blocks. A block refers to a component or module of the model, which may contain multiple layers (such as convolutional layers, pooling layers, fully connected layers, etc.) and possible activation functions. By stacking multiple such blocks, complex deep learning models can be built. Each block is responsible for extracting a specific type of feature or performing a specific data transformation, and the performance of the entire model depends on how these blocks work together. For example, in convolutional neural networks (CNNs), a series of convolutional layers, pooling layers, and fully connected layers can be considered as different blocks that together constitute the entire model.

[0105] Therefore, the role of the sampling model is to predict, based on the current first training image (i.e., the image with some noise already added), how much noise should be added in the next sampling, i.e., how many time steps are needed to obtain high-quality training data for the diffusion model. The first sampling step size predicted by the sampling model refers to the optimal denoising time for the image based on the current state (first starting time).

[0106] 104. The first training image is denoised using a diffusion model based on the first sampling step size to obtain the first sample image at the first sample time.

[0107] The diffusion model is used here to attempt to remove noise from the first training image in order to generate samples that are closer to the original image. For example, when the first sampling step size predicted by the sampling model is 100, the diffusion model denoises the first training image with a step size of 100. In this case, the first sample time corresponding to the first sample image is 900-100=800, which can be understood as the noise level of the first sample image being 80%. As another example, when the first sampling step size predicted by the sampling model is 50, the first sample time corresponding to the first sample image is 900-50=950, which can be understood as the noise level of the first sample image being 95%.

[0108] 105. Add noise to the initial image based on the first sample time to obtain the first desired image.

[0109] Understandably, comparing the similarity between predicted and expected values ​​is a crucial step in evaluating model performance during model training. This step not only helps understand the current accuracy of the model but also guides subsequent model optimization. In this embodiment, after predicting the first noisy image to obtain the first sampling step size, the first sample image is obtained by denoising the first training image based on the first sampling step size. The first sample image can be considered as an intermediate state corresponding to the first sampling time, based on the predicted value at the current training stage, i.e., the process in which the model attempts to recover a clear image from the noise. The initial image is directly denoised to the noisy state corresponding to the first sampling time to obtain the first expected image. The first expected image is equivalent to the expected value in model training and serves as a supervision signal for training, guiding the training of the sampling model. The higher the similarity between the first sample image and the first expected image, the better the sampling model's predictive ability for the sampling effect.

[0110] 106. Train the sampling model based on the loss function term, which indicates the similarity between the first sample image and the first expected image.

[0111] After obtaining the predicted values ​​(first sample image) and expected values ​​(first expected image) from the model training, the sampled model can be trained based on the loss function term, which indicates the similarity between the predicted and expected values. Specific methods for measuring the similarity or difference between the predicted and expected values ​​can include mean squared error, root mean square error, mean absolute error, and R². 2 Fractions, cross-entropy loss, etc.

[0112] Understandably, by repeating the above steps multiple times, the parameters of the sampling model can be gradually optimized. There are many different ways to design and implement the sampling model. For example, it can be a neural network-based model that learns the statistical regularities in the data to predict the distribution and intensity of noise.

[0113] The sampling model training method provided in this application embodiment can predict the image at any intermediate state during the image diffusion process, dynamically allocate the next sampling step size, and use the sampling model as the prediction guide for the diffusion model sampler. It can match different sampling time series for each different input, so as to ensure the quality of diffusion model generation while significantly reducing computation time and resource consumption, and improving the generation efficiency of diffusion model.

[0114] Please refer to Figure 2, which is a flowchart of the training method for the sampling model provided in the embodiments of this application, including:

[0115] 201. Determine the initial image and the first start time;

[0116] 202. Add noise to the initial image based on the first start time to obtain the first training image.

[0117] It is understood that steps 201 and 202 correspond to steps 101 and 102 in the embodiment shown in Figure 1, and will not be described again here.

[0118] 203. Based on the first training image, a sampling prediction is performed by a sampling model to obtain multiple first sampling step sizes. The multiple first sampling step sizes correspond to the first sample image and multiple first expected images at multiple first sample times, respectively. The multiple first sampling step sizes include a first sub-sampling step size and a second sub-sampling step size, wherein the first sub-sampling step size is larger than the second sub-sampling step size.

[0119] Understandably, during the training of a sampling model, multiple different sampling step sizes can be predicted for an image in a given state. By training the sampling model based on the similarity between the predicted and expected values ​​corresponding to multiple different sampling step sizes, the trained sampling model can tend to select the sampling step size with the best effect.

[0120] In one possible implementation, the multiple first sampling steps include a first sub-sampling step and a second sub-sampling step. In this embodiment, during the subsequent similarity comparison process, if the effects corresponding to the two sub-sampling steps are comparable (similar in degree), in order to further improve the sampling efficiency, a supervision method can be added so that the model can choose a larger jump step as much as possible when the effects are comparable. For details, please refer to the following text.

[0121] 204. The first training image is denoised using a diffusion model based on multiple first sampling steps to obtain multiple first sample images at multiple first sample times.

[0122] Understandably, once multiple first sampling step sizes are obtained through prediction, the first training image can be denoised using a diffusion model based on each of these first sampling step sizes, thereby obtaining multiple corresponding first sample images. Since there is a one-to-one correspondence between sample images and sample times, multiple first sample images correspond to multiple different first sample times.

[0123] 205. Noise is added to the initial image based on multiple first sample times to obtain multiple corresponding first desired images.

[0124] It is understandable that when there are multiple first sample times, it is necessary to add noise to the initial image based on each of the multiple sample times to obtain the corresponding multiple first expected images.

[0125] 206. Calculate the similarity between the corresponding multiple first sample images and multiple first expected images.

[0126] Understandably, the purpose of the similarity calculation here is to quantify the difference between the first sample image and the first expected image. The higher the similarity, the better the corresponding first sampling step size is chosen.

[0127] In one possible implementation, when the plurality of first sampling steps includes a first sub-sampling step and a second sub-sampling step, step 206 specifically includes:

[0128] 2061. Calculate the first similarity between the first subsample image and the first sub-expected image. The first subsample image is obtained by denoising the first training image based on the first subsample step size. The first sub-expected image is obtained by adding noise to the initial image based on the first subsample time corresponding to the first subsample image.

[0129] 2062. Calculate the second similarity between the second subsample image and the second subtraining image. The second subsample image is obtained by denoising the second training image based on the second subsampling step size. The second subexpected image is obtained by adding noise to the initial image based on the second subsample time corresponding to the second subsample image.

[0130] 2063. If the difference between the first similarity and the second similarity is less than a preset value, then the first similarity is increased based on the preset reward function.

[0131] In this embodiment of the application, in order to encourage the sampling model to choose a larger step size when the effect is comparable, a reward function is designed to adjust the balance between efficiency and effect of the sampling model. That is, when the first sub-sampling step size is larger than the second sub-sampling step size and the difference between the first similarity and the second similarity is small, the sampling model can be more inclined to choose a larger sampling step size by further increasing the first similarity.

[0132] Specifically, the reward function can indicate a first weight and a second weight, where the first weight corresponds to the first subsampling step size and the second weight corresponds to the second subsampling step size, and the first weight is greater than the second weight.

[0133] The reward function can be implemented using the following formula:

[0134]

[0135] This formula takes into account the relationship between three factors. Among them, A represents the set of possible step sizes (multiple first sampling step sizes). i σ represents the currently chosen step size strategy (first subsampling step size or second subsampling step size), while σ is the softmax function used to transform this expression into possibilities for each choice. To balance the trade-off between sampling speed and quality, a hyperparameter α is introduced. As α increases, the sampling model tends to plan for larger step sizes. Typically, α is set to 1.

[0136] 207. The first sample time corresponding to the highest similarity is determined as the second starting time, and noise is added to the initial image based on the second starting time to obtain the second training image. The second training image corresponds to the second sample image and the second expected image under the second sampling step and the second sample time.

[0137] Understandably, in machine learning and deep learning, especially when dealing with sequence data or time series analysis (such as natural language processing, speech recognition, and time series prediction), the training process of classification models may encounter a problem: because classification models can only be supervised by the results at the current time step, they can easily get trapped in local optima. This is because most supervised learning models, including classification models, are trained based on "input-output" pairs. For classification tasks, this means that the model receives an input (such as the first training image) and attempts to predict a class label. During training, the model adjusts its internal parameters by comparing the difference between its prediction (the first sample image) and the actual label (the first expected image) to reduce this difference (usually by minimizing a loss function). In optimization problems, a local optimum is a solution that is optimal within its neighborhood but not globally optimal. To address this issue, the training method provided in this application embodiment determines the first sample time as the second start time of the second sample acquisition process based on the similarity calculation result of step 206. The selection of the second start time is preferably the first sample time corresponding to the result with the highest similarity among multiple similarity calculation results in the first sample acquisition process (i.e., steps 201 to 206).

[0138] In this embodiment, joint supervision at multiple time points is used to encourage the model to learn the global optimal solution, which can avoid getting trapped in local optima.

[0139] It is understandable that the process and principle of using the second starting time as a reference to predict the step size through the sampling model are similar to those using the first starting time as a reference. Therefore, the second training image can specifically correspond to multiple second sampling step sizes, and the multiple second sampling step sizes correspond to the second sample images and multiple second expected images under multiple second sample times, respectively.

[0140] In one possible implementation, step 207 specifically includes:

[0141] 2071. The time of the first sample corresponding to the highest similarity is determined as the second starting time;

[0142] 2072. Add noise to the initial image based on the second start time until the number of predictions of the sampling model reaches a preset value.

[0143] In the implementation method provided in this application embodiment, considering that too much supervision information at too many times will lead to increased training costs and unstable training, the number of predictions for each training session can be limited. When the number of predictions reaches a preset value (e.g., 5 times), the subsequent step of training the sampling model according to the loss function is executed.

[0144] In another possible implementation, prior to step 207, the following is also included:

[0145] 200. Divide the total step size into multiple time intervals, where the total step size is the step size used to spread the initial image to a completely noisy image;

[0146] Step 207 specifically includes:

[0147] 2073. Determine the target time period to which the first start time belongs; the target time period may belong to multiple time periods.

[0148] 2074. If the second start time is not less than the minimum time of the target time period, then add noise to the initial image based on the second start time.

[0149] In the implementation method provided in this application embodiment, the total step size is pre-divided. For example, taking a total step size of 1000 as an example, it can be divided into 5 equally divided time periods, namely [0,200), [200,400), [400,600), [600,800) and [800,1000].

[0150] After determining the first start time in step 201, the time period to which the first start time belongs is further determined and designated as the target time period. For example, if the first start time is 8:50, according to the time period division in the example above, the first start time belongs to the time period [8:00, 10:00], and this time period is the target time period.

[0151] Assume the sampling model obtains three first sample steps based on the first training image, with step sizes of 20, 40, and 60, corresponding to first sample times of 830, 810, and 790, respectively. Calculate the similarity between the first sample image and the first desired image in these three cases. If the similarity is highest at first sample time 810, and since 810 is still within the time range [800, 1000], the second sample time is set to 810, and the subsequent sampling process continues. If the similarity is highest at first sample time 790, but 790 is already less than the minimum value (800) within the time range [800, 1000], then sampling ends, and the subsequent steps of training based on the loss function are executed.

[0152] Understandably, both of the above steps 207 can reduce the learning complexity of the sampling model and achieve effective and stable model training.

[0153] 208. The sampling model is trained based on the loss function term, which specifically indicates the similarity between the corresponding sample image and the expected image. The sample image includes the first sample image and the second sample image, and the expected image includes the first expected image and the second expected image.

[0154] It is understandable that when the sampling process involves multiple rounds, the loss function specifically refers to the similarity between each sample image and its corresponding expected image under multiple rounds of sampling. Taking two rounds of sampling as an example, the sample images include the first sample image and the second sample image, and the expected images include the first expected image and the second expected image. It is also understandable that when the number of samplings is even greater, it may include a third, fourth, etc., which will not be elaborated here.

[0155] The sampling model training method provided in this application can perceive the input and all intermediate states of the corresponding samples during sampling, and select the optimal denoised time series in real time, which is very flexible. At the same time, the lightweight classifier will hardly bring any additional computational burden during sampling. By setting the hyperparameters of the reward function, the classification model can be flexibly adjusted between efficiency and accuracy. In addition, the method can generate pseudo-labels for the prediction results at the current time and then generate pseudo-labels for the prediction results at the next time (this process can be repeated many times). By using joint supervision at multiple time points, the model is encouraged to learn the global optimum and avoid getting trapped in local optima. However, too much supervision information at too many time points will increase the training cost and also lead to training instability. Therefore, a segmented learning mechanism is proposed to reduce the learning complexity and effectively stabilize model training.

[0156] Based on the training method of the above sampling model, this application also provides an image processing method. Please refer to Figure 3, which is a flowchart of the image processing method provided in the embodiment of this application, including:

[0157] 301. Obtain the first noisy image;

[0158] 302. The first sampling step size is predicted based on the first noisy image by the sampling model, and the sampling model is trained based on the training method of the sampling model corresponding to Figure 1 or Figure 2;

[0159] 303. The first noisy image is denoised using a diffusion model based on the first sampling step size to obtain the second noisy image.

[0160] It is understood that the embodiments of this application are an image processing method, specifically a method for restoring noisy images based on a diffusion model. Since the diffusion model needs to sample according to a certain time step during the process of restoring noisy images, the sampling process can be predicted by the sampling model trained in the above embodiments.

[0161] The first noisy image can be completely random noise or an image between a clear image and a noisy image, obtained by adding noise to an existing image. The task of the sampling model is to predict a sampling step size (or a time step), i.e., the first sampling step size, which guides the denoising process of the first noisy image. The first sampling step size can be regarded as a control parameter, affecting the speed or path of the first noisy image gradually transitioning from a noisy state to a clear state. After predicting the first sampling step size, the diffusion model can denoise the first noisy image based on this first sampling step size to obtain a second noisy image, which is usually a clearer image compared to the first noisy image.

[0162] In one possible implementation, after step 303, the method further includes:

[0163] 304. Predict the second sampling step size based on the second noisy image using a sampling model;

[0164] 305. The second image is obtained by denoising the second noisy image using a diffusion model based on the second sampling step size.

[0165] It is understandable that the sampling model trained using the above-described sampling model training method can predict the sampling step size for images in any intermediate state (between completely noisy and completely clear images). Therefore, after obtaining the second noisy image, the sampling model can also predict the second sampling step size based on the second noisy image, so that the diffusion model can denoise the second noisy image based on the second sampling step size to obtain the third image. Generally speaking, the third image is a clearer image relative to the second noisy image; the third image can be either a noisy image or a clear image.

[0166] For ease of understanding, please refer to Figure 4, which is a schematic diagram of the application of the sampling model obtained by the training method of the sampling model provided in the embodiment of this application.

[0167] First, a series of noisy images are identified. Then, the sampling model predicts the first sampling step size as t0, corresponding to a sampling time of s0. A diffusion model is then used to denoise the noisy images based on t0, resulting in the noisy image p0 at time s0. After obtaining the noisy image p0, the sampling model is used again to predict the second sampling step size as 10. The noisy image p0 at time s0 is then denoised based on a step size of 10, resulting in the noisy image p1 at time s1. This process is repeated for each noisy image. i-1 Noisy image p over time i-1 Prediction is performed using a sampling model, resulting in a sampling step size of 50. The diffusion model then applies a step size of 50 to the noisy image p. i-1 Denoising is performed to obtain si Noisy image p over time i This continues until the noisy image is completely denoised into a clear image.

[0168] The effectiveness of the sampling model obtained by the training method of the sampling model provided in the embodiments of this application will be introduced next.

[0169] Please refer to Figures 5 and 6. Figure 5 is a schematic diagram comparing the performance of the sampling method optimized by the sampling model provided in this application on two datasets. Figure 6 is a schematic diagram comparing the performance of multiple different sampling strategies.

[0170] As shown in Figure 5, in the CIFAR-10 and LSUN-bedroom datasets, each row of the table represents a specific parameter or metric. Dataset refers to the dataset name, including CIFAR-10 and LSUN-Bedroom; Method refers to the sampling method, including DDIM+RASS and DPM-Solver+RASS. RASS refers to the sampling model trained based on the training methods in the corresponding embodiments of Figures 1 and 2 of this application. This sampling model serves as the prediction guide for existing DDIM and DPM-Solver samplers, enabling matching different sampling time step sequences for each different input; α refers to the hyperparameter, including 1.5, 1, and 0.5; step is the number of sampling steps; FID refers to the metric for measuring the quality of the generated image; a lower FID value indicates higher image quality; FID_Ref and Step_Ref refer to the reference values ​​for the FID score and the number of sampling steps, respectively, under the existing conditions (without using the sampling model as the prediction guide for the sampler); Speed ​​refers to the speed factor for each method, representing the performance improvement relative to the reference value.

[0171] The left image in Figure 6 compares the sampling performance of the DDIM, DPM-Solver, DDIM+RASS, and DPM-Solver+RASS methods on the CIFAR-10 and LSUN-bedroom datasets. The right image compares the images generated by the diffusion model under the DDIM and DDIM+RASS methods.

[0172] As shown in Figures 5 and 6, on the CIFAR-10 and LSUN-bedroom datasets for unconditional generation tasks, subjective results show that, compared with existing methods, the solution provided by the embodiments of this application can generate more realistic images; objective results show that, under the same FID, the solution provided by the embodiments of this application can achieve a speedup ratio of 1.25 to 1.4, significantly accelerating the sampling process of the diffusion model.

[0173] On the MS-COCO 2017 dataset for text image generation, objective results show that, compared with existing methods, the solution provided in this application significantly outperforms the DDIM / DPM-Solver with 35 steps in all metrics at 25 steps or even 15 steps, achieving a speedup of over 1.4, thus proving the effectiveness of the solution provided in this application.

[0174] On the CIFAR-10 dataset, an unconditional generation task, the solution provided in this application was experimentally compared with the prior art method three. Experimental results show that the solution provided in this application significantly outperforms the prior art method three, and the training cost can be reduced by more than four times, demonstrating the effectiveness of the solution provided in this application.

[0175] This application also provides a training device for a sampling model. Please refer to Figure 7, which is a schematic diagram of the structure of the training device for a sampling model provided in an embodiment of this application, including:

[0176] The determination module 701 is used to determine the initial image and the first start time;

[0177] The noise-adding module 702 is used to add noise to the initial image based on the initial time to obtain the first training image;

[0178] Prediction module 703 is used to perform sampling prediction based on the first training image using a sampling model to obtain the first sampling step size;

[0179] The denoising module 704 is used to denoise the first training image based on the first sampling step size using a diffusion model to obtain the first sample image at the first sample time.

[0180] The noise-adding module 702 is also used to add noise to the initial image based on the first sample time to obtain the first desired image;

[0181] Training module 705 is used to train the sampling model based on a loss function term, which indicates the similarity between the first sample image and the first desired image.

[0182] The sampling model training device provided in this application embodiment is used to train the sampling model, so that the sampling model can predict the image at any intermediate state during the image diffusion process, dynamically allocate the next sampling step size, and use the sampling model as the prediction guide for the diffusion model sampler. Different sampling time series can be matched for each different input, so as to ensure the quality of diffusion model generation, while greatly reducing the computation time and resource consumption, and improving the generation efficiency of diffusion model.

[0183] In one possible implementation, the prediction module 703 is specifically used to perform sampling prediction based on the first training image through a sampling model to obtain multiple first sampling step sizes, the multiple first sampling step sizes corresponding to the first sample image and multiple first expected images under multiple first sample times respectively;

[0184] It also includes: a calculation module, used to calculate the similarity between the corresponding multiple first sample images and multiple first expected images; to determine the first sample time corresponding to the highest similarity as the second start time, and to add noise to the initial image based on the second start time to obtain the second training image, wherein the second training image corresponds to the second sample image and the second expected image under the second sampling step size and the second sample time.

[0185] Correspondingly, the loss function term specifically indicates the similarity between the corresponding sample image and the expected image. The sample image includes the first sample image and the second sample image, and the expected image includes the first expected image and the second expected image.

[0186] In the embodiments of this application, during the training process of the sampling model, multiple different sampling step lengths can be predicted for an image in a certain state. By training the sampling model based on the similarity between the predicted value and the expected value corresponding to multiple different sampling step lengths, the trained sampling model can tend to select the sampling step length with the best effect.

[0187] In one possible implementation, the multiple first sampling steps include a first sub-sampling step and a second sub-sampling step. In this embodiment, during the subsequent similarity comparison process, if the effects corresponding to the two sub-sampling steps are comparable (similar in degree), in order to further improve the sampling efficiency, a supervision method can be added so that the model can choose a larger jump step as much as possible when the effects are comparable. For details, please refer to the following text.

[0188] In one possible implementation, the second training image specifically corresponds to multiple second sampling steps, and the multiple second sampling steps respectively correspond to the second sample image and multiple second expected images at multiple second sample times.

[0189] It is understandable that the process and principle of using the second starting time as a reference to predict the step size through the sampling model are similar to those of using the first starting time as a reference.

[0190] In one possible implementation, the calculation module is specifically used to determine the first sample time corresponding to the highest similarity as the second starting time; and to add noise to the initial image based on the second starting time until the number of predictions of the sampling model reaches a preset value.

[0191] In this embodiment of the application, considering that too much supervision information at any time will lead to increased training costs and training instability, the number of predictions for each training session can be limited. When the number of predictions reaches a preset value (e.g., 5 times), the subsequent step of training the sampling model according to the loss function is executed.

[0192] In another possible implementation, the determining module 701 is also used to divide the total step size into multiple time periods, the total step size being the step size for spreading the initial image to a completely noisy image;

[0193] The denoising module 704 is specifically used to determine the target time period to which the first start time belongs, and the target time period belongs to multiple time periods; if the second start time is not less than the minimum time of the target time period, then noise is added to the initial image based on the second start time.

[0194] In this embodiment, the total step size is pre-divided. After determining the first start time, the time period to which the first start time belongs is further determined and designated as the target time period. If the second start time exceeds the target time period, samples are no longer acquired. This also reduces the learning complexity of the sampling model and achieves effective and stable model training.

[0195] In one possible implementation, the prediction module 703 is specifically used to perform sampling prediction based on the first training image through a sampling model to obtain multiple first sampling step sizes, the multiple first sampling step sizes including a first sub-sampling step size and a second sub-sampling step size, the first sub-sampling step size being larger than the second sub-sampling step size.

[0196] The calculation module is specifically used to calculate the first similarity between the first subsample image and the first sub-expected image. The first subsample image is obtained by denoising the first training image based on the first subsampling step size, and the first sub-expected image is obtained by adding noise to the initial image based on the first subsample time corresponding to the first subsample image. The module also calculates the second similarity between the second subsample image and the second subtraining image. The second subsample image is obtained by denoising the second training image based on the second subsampling step size, and the second sub-expected image is obtained by adding noise to the initial image based on the second subsample time corresponding to the second subsample image. If the difference between the first similarity and the second similarity is less than a preset value, the first similarity is increased based on a preset reward function.

[0197] In this embodiment of the application, in order to encourage the sampling model to choose a larger step size when the effect is comparable, a reward function is designed to adjust the balance between efficiency and effect of the sampling model. That is, when the first sub-sampling step size is larger than the second sub-sampling step size and the difference between the first similarity and the second similarity is small, the sampling model can be more inclined to choose a larger sampling step size by further increasing the first similarity.

[0198] In one possible implementation, the reward function indicates a first weight and a second weight, the first weight corresponding to a first subsampling step size and the second weight corresponding to a second subsampling step size; the first weight is greater than the second weight.

[0199] This application also provides an image processing apparatus. Please refer to FIG8, which is a schematic diagram of the structure of the image processing apparatus provided in the embodiment of this application, including:

[0200] Acquisition module 801 is used to acquire the first noisy image;

[0201] The sampling module 802 is used to predict the first sampling step size based on the first noisy image through a sampling model. The sampling model is trained based on the training method of the sampling model corresponding to the embodiment in Figure 1 or Figure 2.

[0202] The denoising module 803 is used to denoise the first noisy image based on the first sampling step size.

[0203] The image processing apparatus provided in this application embodiment can effectively improve sampling efficiency by using a trained sampling model as a sampler to predict the sampling step size of noisy images.

[0204] In one possible implementation,

[0205] The sampling module 802 is also used to predict the second sampling step size based on the second noisy image through the sampling model;

[0206] The denoising module 803 is also used to denoise the second noisy image based on the second sampling step size using a diffusion model to obtain the third image.

[0207] It is understandable that the sampling model trained using the above sampling model training method can predict the sampling step size for images in any intermediate state (between completely noisy images and completely clear images).

[0208] This application also provides a computing device. Please refer to FIG9, which is a schematic diagram of the structure of the computing device provided in the embodiment of this application. The computing device can be used to execute computer programs or computer instructions in memory to perform the methods in the embodiments shown in FIG1 to FIG3. Refer to the relevant descriptions in the above method embodiments.

[0209] The communication device includes a processor 901. Optionally, the communication device may also include a memory 902 and a transceiver 903.

[0210] This application also provides a computer program product including instructions that, when run on a computer, cause the computer to perform the methods shown in the embodiments of Figures 1 to 3 above.

[0211] This application also provides a computer-readable storage medium including computer instructions that, when executed on a computer, cause the computer to perform the methods shown in the embodiments of Figures 1 to 3 above.

[0212] This application also provides a chip device, including a processor for connection to a memory, and for calling a program stored in the memory so that the processor executes the method of the embodiment shown in FIG3 above.

[0213] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of a program that controls the method of the embodiment shown in Figure 6. The memory mentioned above can be read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0214] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0215] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0216] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0217] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0218] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A training method for a sampling model, characterized in that, include: Determine the initial image and the first start time; Noise is added to the initial image based on the first start time to obtain the first training image; The first sampling step size is obtained by performing sampling prediction based on the first training image using a sampling model; The first training image is denoised using a diffusion model based on the first sampling step size to obtain the first sample image at the first sample time. Noise is added to the initial image based on the first sample time to obtain a first desired image; the sampling model is trained according to a loss function term, which indicates the similarity between the first sample image and the first desired image.

2. The training method according to claim 1, characterized in that, The step of obtaining a first sampling step size by sampling prediction based on the first training image using a sampling model includes: obtaining multiple first sampling step sizes by sampling prediction based on the first training image using a sampling model, wherein the multiple first sampling step sizes correspond to the first sample images and multiple first expected images under multiple first sample times respectively; after adding noise to the initial image based on the first sample time to obtain the first expected image, the step further includes: calculating the similarity between the corresponding multiple first sample images and multiple first expected images; determining the first sample time corresponding to the highest similarity as the second starting time, and adding noise to the initial image based on the second starting time to obtain a second training image, wherein the second training image corresponds to the second sample image and the second expected image under the second sampling step size and the second sample time; the loss function term specifically indicates the similarity between the corresponding sample image and the expected image, wherein the sample image includes the first sample image and the second sample image, and the expected image includes the first expected image and the second expected image.

3. The training method according to claim 2, characterized in that, The second training image specifically corresponds to multiple second sampling steps, and the multiple second sampling steps respectively correspond to the second sample image and multiple second expected images at multiple second sample times.

4. The training method according to claim 2 or 3, characterized in that, The step of determining the first sample time corresponding to the highest similarity as the second starting time and adding noise to the initial image based on the second starting time specifically includes: determining the first sample time corresponding to the highest similarity as the second starting time; adding noise to the initial image based on the second starting time until the number of predictions of the sampling model reaches a preset value.

5. The training method according to claim 2 or 3, characterized in that, Before determining the first sample time corresponding to the highest similarity as the second starting time and adding noise to the initial image based on the second starting time, the method further includes dividing the total step size into multiple time periods, where the total step size is the step size for spreading the initial image to a completely noisy image. Specifically, determining the first sample time corresponding to the highest similarity as the second starting time and adding noise to the initial image based on the second starting time includes: determining the target time period to which the first starting time belongs, where the target time period belongs to the multiple time periods; if the second starting time is not less than the minimum time of the target time period, then adding noise to the initial image based on the second starting time.

6. The training method according to any one of claims 1 to 4, characterized in that, The step of obtaining a first sampling step size by sampling prediction based on the first training image using a sampling model includes: obtaining multiple first sampling step sizes by sampling prediction based on the first training image using a sampling model, wherein the multiple first sampling step sizes include a first sub-sampling step size and a second sub-sampling step size, and the first sub-sampling step size is larger than the second sub-sampling step size; before training the sampling model according to the loss function term, the step further includes: calculating a first similarity between a first sub-sample image and a first sub-expected image, wherein the first sub-sample image is obtained by denoising the first training image based on the first sub-sampling step size, and the first sub-expected image is obtained by adding noise to the initial image based on the first sub-sample time corresponding to the first sub-sample image; calculating a second similarity between a second sub-sample image and a second sub-training image, wherein the second sub-sample image is obtained by denoising the second training image based on the second sub-sampling step size, and the second sub-expected image is obtained by adding noise to the initial image based on the second sub-sample time corresponding to the second sub-sample image; if the difference between the first similarity and the second similarity is less than a preset value, then the first similarity is increased based on a preset reward function.

7. The training method according to claim 6, characterized in that, The reward function indicates a first weight and a second weight, the first weight corresponding to the first sub-sampling step size, and the second weight corresponding to the second sub-sampling step size; the first weight is greater than the second weight.

8. An image processing method, characterized in that, include: Acquire the first noisy image; The sampling model predicts the first sampling step size based on the first noisy image using a sampling model, wherein the sampling model is trained based on the training method of the sampling model according to any one of claims 1 to 7; The first noisy image is denoised using a diffusion model based on the first sampling step size to obtain a second noisy image.

9. The method according to claim 8, characterized in that, After denoising the first noisy image using a diffusion model based on the first sampling step size to obtain a first image, the method further includes: predicting a second sampling step size using the sampling model based on the second noisy image; and denoising the second noisy image using the diffusion model based on the second sampling step size to obtain a third image.

10. A training device for a sampling model, characterized in that, include: The determination module is used to determine the initial image and the first start time; The noise-adding module is used to add noise to the initial image based on the initial time to obtain a first training image; the prediction module is used to perform sampling prediction based on the first training image using a sampling model to obtain a first sampling step size; the denoising module is used to denoise the first training image based on the first sampling step size using a diffusion model to obtain a first sample image at a first sample time. The noise-adding module is further configured to add noise to the initial image based on the first sample time to obtain a first desired image; A training module is used to train the sampling model based on a loss function term, which indicates the similarity between the first sample image and the first desired image.

11. An image processing apparatus, characterized in that, include: The acquisition module is used to acquire the first noisy image; A sampling module is used to predict a first sampling step size based on the first noisy image using a sampling model, wherein the sampling model is trained based on the training method of the sampling model according to any one of claims 1 to 7; A denoising module is used to denoise the first noisy image based on the first sampling step size.

12. A computing device, characterized in that, Includes a processor; the processor is configured to execute a computer program or computer instructions in memory to perform a training method for a sampling model as described in any one of claims 1 to 7, or an image processing method as described in claim 8 or 9.

13. The computing device according to claim 12, characterized in that, It also includes the memory.

14. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device, the computing device performs the training method of the sampling model as described in any one of claims 1 to 7, or the image processing method as described in claim 8 or 9.

15. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a computing device, enable the computing device to perform a training method for a sampling model as described in any one of claims 1 to 7, or an image processing method as described in claim 8 or 9.

Images