Diffusion model processing method and device, picture processing method and device

Abstract

Embodiments of the present specification provide a diffusion model processing method and device, a picture processing method and device, wherein the diffusion model processing method comprises determining a time step set of a diffusion model and a time step interval corresponding to the time step set; determining a first time step from the time step set and determining a target time step corresponding to the first time step according to the time step interval; inputting a noisy picture corresponding to the first time step and the target time step into the diffusion model to obtain predicted noise corresponding to the noisy picture; and processing the diffusion model according to target noise corresponding to the noisy picture and the predicted noise. The method divides the time step set into time step intervals, and during subsequent diffusion model training, the diffusion model shares time step conditions within a time step interval, i.e., the first time step shares the target time step within the corresponding time step interval, thereby reducing the time step condition, greatly reducing the training burden, and improving the model training performance.

Description

TECHNICAL FIELD

[0001] Embodiments of the present specification relate to the technical field of computer technology, and particularly relate to a diffusion model processing method and device, a picture processing method and device, a computing device, and a computer readable storage medium. BACKGROUND

[0002] A diffusion model is a generative model that can gradually destroy a picture distribution into Gaussian noise by constructing a Markov chain, and generate a picture by gradually denoising the reverse distribution learned by a network. The diffusion model has achieved very amazing results on various tasks, including but not limited to the multi-modal generation task of generating images from text.

[0003] However, the traditional diffusion model usually learns all single-step transition probabilities through a network, and controls the learning of different probabilities through time step conditions. This way causes a heavy training burden, resulting in insufficient model capacity and poor training effect. SUMMARY

[0004] Therefore, the embodiments of the present specification provide a diffusion model processing method. One or more embodiments of the present specification also relate to a diffusion model processing device, a picture processing method, a picture processing device, a computing device, a computer readable storage medium, and a computer program to solve the technical defects in the prior art.

[0005] According to a first aspect of the embodiments of the present specification, a diffusion model processing method is provided, comprising:

[0006] determining a time step set of a diffusion model and a time step interval corresponding to the time step set;

[0007] determining a first time step from the time step set and determining a target time step corresponding to the first time step according to the time step interval, wherein the first time step is any time step in the time step set;

[0008] inputting a noisy picture corresponding to the first time step and the target time step into the diffusion model to obtain predicted noise corresponding to the noisy picture;

[0009] processing the diffusion model according to target noise corresponding to the noisy picture and the predicted noise.

[0010] According to a second aspect of the embodiments of the present specification, a diffusion model processing device is provided, comprising:

[0011] an interval division module configured to determine a time step set of a diffusion model and a time step interval corresponding to the time step set;

[0012] a target time step determination module configured to determine a first time step from the set of time steps, and determine a target time step corresponding to the first time step according to the time step interval, wherein the first time step is any time step in the set of time steps;

[0013] a first model prediction module configured to input the noisy picture corresponding to the first time step and the target time step into a diffusion model, and obtain predicted noise corresponding to the noisy picture;

[0014] a model processing module configured to process the diffusion model according to target noise corresponding to the noisy picture and the predicted noise.

[0015] According to a third aspect of an embodiment of the present specification, a picture processing method is provided, comprising:

[0016] determining a target noisy picture, inputting the target noisy picture into a diffusion model, and obtaining predicted noise corresponding to the target noisy picture;

[0017] determining a denoised target picture according to the target noisy picture and the predicted noise corresponding to the target noisy picture,

[0018] wherein the diffusion model is obtained by the diffusion model processing method.

[0019] According to a fourth aspect of an embodiment of the present specification, a picture processing apparatus is provided, comprising:

[0020] a second model prediction module configured to determine a target noisy picture, input the target noisy picture into a diffusion model, and obtain predicted noise corresponding to the target noisy picture;

[0021] a target picture determination module configured to determine a denoised target picture according to the target noisy picture and the predicted noise corresponding to the target noisy picture,

[0022] wherein the diffusion model is obtained by the diffusion model processing method.

[0023] According to a fifth aspect of an embodiment of the present specification, a computing device is provided, comprising:

[0024] a memory and a processor;

[0025] the memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions, and the computer executable instructions, when executed by the processor, implement the steps of the diffusion model processing method or the picture processing method.

[0026] According to a sixth aspect of an embodiment of the present specification, a computer-readable storage medium is provided, which stores computer-executable instructions that, when executed by a processor, implement the steps of the diffusion model processing method or the picture processing method described above.

[0027] According to a seventh aspect of an embodiment of the present specification, a computer program is provided, which, when executed in a computer, causes the computer to perform the steps of the diffusion model processing method or the picture processing method described above.

[0028] One embodiment of the present specification implements a diffusion model processing method, comprising determining a time step set of a diffusion model and a time step interval corresponding to the time step set; determining a first time step from the time step set and determining a target time step corresponding to the first time step according to the time step interval, wherein the first time step is any time step in the time step set; inputting a noisy picture corresponding to the first time step and the target time step into the diffusion model to obtain predicted noise corresponding to the noisy picture; and processing the diffusion model according to target noise corresponding to the noisy picture and the predicted noise.

[0029] Specifically, the diffusion model processing method divides the time step set into time step intervals, and during subsequent diffusion model training, the diffusion model shares time step conditions within a time step interval, i.e., the first time step shares the target time step within its corresponding time step interval, which reduces the time step condition and greatly reduces the training burden, improving the model training performance. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 is a structural diagram of a denoising diffusion probability model provided by one embodiment of the present specification;

[0031] Figure 2 is a specific implementation scenario diagram of a diffusion model processing method provided by one embodiment of the present specification;

[0032] Figure 3 is a flowchart of a diffusion model processing method provided by one embodiment of the present specification;

[0033] Figure 4 is a flowchart of a picture processing method provided by one embodiment of the present specification;

[0034] Figure 5 is a structural diagram of a diffusion model processing device provided by one embodiment of the present specification;

[0035] Figure 6 is a structural diagram of a picture processing device provided by one embodiment of the present specification;

[0036] Figure 7 is a structural block diagram of a computing device provided by an embodiment of the present specification. DETAILED DESCRIPTION

[0037] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present specification. However, the present specification can be practiced without the specific details, other than in the examples described herein, and it is understood that the scope of the present specification is not limited to the details below.

[0038] The terminology used in one or more embodiments of the present specification is for the purpose of describing particular embodiments only and is not intended to be limiting of one or more embodiments of the present specification. As used in one or more embodiments of the present specification and the accompanying claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in one or more embodiments of the present specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0039] It is to be understood that although the terms first, second, etc. can be employed in one or more embodiments of the present specification, these terms are used to distinguish one from another only and are not necessarily used to describe a number sequence. For example, without departing from the scope of one or more embodiments of the present specification, first can be termed second, and similarly, second can be termed first. Depending on the context, the word "if' as used herein can be interpreted to mean "when" or "in response to determining."

[0040] First, the noun terms related to one or more embodiments of the present specification are explained.

[0041] Diffusion Model: an image generation method for generating high-quality pictures of multiple categories.

[0042] Interval division: dividing time steps in a Markov chain of a diffusion model into multiple intervals.

[0043] Denoising Diffusion Probabilistic Model: DDPM, Denoising Diffusion Probabilistic Model.

[0044] In the specification, a diffusion model processing method is provided. One or more embodiments of the specification also relate to a diffusion model processing apparatus, an image processing method, an image processing apparatus, a computing device, a computer-readable storage medium, and a computer program, which are described in detail in the following embodiments.

[0045] Referring to Figure 1 , Figure 1 A structural diagram of a denoising diffusion probability model according to an embodiment of the specification is shown.

[0046] In practical applications, the basic idea of the denoising diffusion probability model is to construct a Markov chain, and model the transition probabilities of all time steps through a network, where the time step condition is used as the conditional input of the network.

[0047] Figure 1 In x, t can be understood as an image, and t can be understood as a time step. Then x t , that is, the image at the t-th time step. If the denoising diffusion probability model is applied to the image denoising scene, the x t , that is, the noisy image at the t-th time step, and x t-1 , that is, the noisy image at the t-th time step; and x0 can be understood as the image obtained by completely denoising the last noisy image.

[0048] Referring to Figure 2 , Figure 2 A specific implementation scenario diagram of a diffusion model processing method according to an embodiment of the specification is shown.

[0049] Figure 2 The cloud-side device 202 and the end-side device 204 are included in the system, where the cloud-side device 202 can be understood as a cloud server, and of course, in another implementation scheme, the cloud-side device 202 can also be replaced by a physical server; the end-side device 204 includes but is not limited to a desktop computer, a notebook computer, etc.; for ease of understanding, in the embodiments of the specification, the cloud-side device 202 is taken as a cloud server, and the end-side device 204 is taken as a notebook computer for detailed description.

[0050] The diffusion model processing method provided in the embodiments of the specification is applied to the image denoising scene and described in detail.

[0051] In specific implementation, the diffusion model training is performed on the cloud-side device 202, where the diffusion model can be understood as a time step condition reduction diffusion model (TSDM).

[0052] As shown in Figure 2 ,Figure 2 For the specific structure of the diffusion model, please refer to [link / reference]. Figure 1 The structure of the denoising diffusion probability model in [the model].

[0053] Figure 1 In the denoising diffusion probability model, a network is needed. θ (x t ,t), modeling T transition probabilities, where t is the time step condition, used to indicate which transition probability the network is currently modeling.

[0054] In practical applications, denoising diffusion probability models require a sufficiently large number of diffusion steps (t) to completely destroy the image signal (i.e., add noise). Therefore, the number of transition probabilities that a single network needs to model is usually very large, which leads to a heavy training burden on the network (i.e., the denoising diffusion probability model). To reduce the training burden, the number of diffusion steps can be reduced, but this also increases the risk of [unclear - possibly "x"]. T The signal-to-noise ratio was not low enough, which caused a significant drop in the final sampling quality.

[0055] Figure 2 The diffusion model provided in the embodiments of this specification divides the diffusion steps into intervals, and divides multiple diffusion steps into one interval. By keeping the number of diffusion steps unchanged, the diffusion model is trained by reducing only the number of time step conditions (i.e., t), thereby reducing the number of transition probabilities that the network needs to model and thus alleviating the burden on the network.

[0056] When the edge device 204 needs to use the diffusion model, it can call the diffusion model trained by the cloud device 202 for functional use. In addition, if the edge device 204 has sufficient computing resources and computing power, the diffusion model trained in the cloud device 202 can also be deployed on the edge device 204. The specific deployment implementation depends on the actual application and is not limited here.

[0057] The diffusion model processing method provided in the embodiments of this specification divides the time step set into time step intervals. When the diffusion model is trained subsequently, the diffusion model shares time step conditions within a time step interval. That is, the first time step shares the target time step within its corresponding time step interval, which reduces the time step conditions, greatly reduces the training burden, and improves the model training performance.

[0058] See Figure 3 , Figure 3 A flowchart of a diffusion model processing method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0059] Step 302: determine a time step set of the diffusion model and a time step interval corresponding to the time step set.

[0060] The diffusion model can be understood as the diffusion model (TSDM) of the time step condition in the above embodiment; the time step set includes a plurality of time steps, and a single time step can be understood as T, T-1, etc. in the above embodiment, which can also be understood as the diffusion step number described in the above embodiment, for example, if the diffusion step number is 10, then it can be understood that the time step set includes 10 time steps. Figure 1

[0061] In order to reduce the number of time step conditions in subsequent diffusion model training, interval division can be performed on the time steps in the time step set. The specific implementation manner is as follows:

[0062] The determination of the time step set of the diffusion model and the time step interval corresponding to the time step set includes:

[0063] The time step set of the diffusion model is determined, and interval division is performed on the time steps in the time step set according to a preset division condition, to obtain the time step interval corresponding to the time step set.

[0064] The preset division condition can be set according to actual application, and the embodiments of the present specification do not make any limitation thereon, for example, the preset division condition can be understood as dividing every 50 time steps into a time step interval.

[0065] For example, the time step set of the diffusion model includes 1000 time steps, and the preset division condition is to divide every 50 time steps into a time step interval.

[0066] Therefore, the determination of the time step set of the diffusion model and the interval division of the time steps in the time step set according to the preset division condition to obtain the time step interval corresponding to the time step set can be understood as the determination of the time step set of the diffusion model (1000 time steps), the interval division according to the preset division condition (dividing every 50 time steps into a time step interval), and the time step interval corresponding to the time step set is 20 time step intervals. When interval division is performed on the time steps in the time step set, interval division needs to be performed according to the time sequence, so the time steps in each time step interval corresponding to the time step set are also sorted according to the time sequence. For example, the first time step interval includes 0-50 time steps arranged according to the time sequence.

[0067] ​Step 304: determining a first time step from the time step set, and determining a target time step corresponding to the first time step according to the time step interval, wherein the first time step is any time step in the time step set.

[0068] Specifically, the first time step can be understood as any one of the time steps in the time step set, such as the first time step, the second time step, the third time step, or the nth time step, etc.

[0069] After determining the first time step from the time step set, the target time step corresponding to the first time step can be determined according to the time step interval corresponding to the time step set.

[0070] In practical applications, since the time steps in the time step set are divided into intervals according to the preset division condition, each time step in the time step set has its corresponding time step interval. Therefore, any one of the time steps in the time step set can determine its corresponding target time step from the time step interval corresponding to it.

[0071] In specific implementation, the time step corresponding to the interval endpoint of the time step interval corresponding to each time step can be used as the target time step corresponding to each time step. Then, when training the diffusion model, the target time step corresponding to the interval endpoint of the time step interval corresponding to each time step can be used as the time step condition, so as to reduce the time step condition when training the diffusion model, and improve the training efficiency and training effect of the diffusion model. The specific implementation mode is as follows:

[0072] The determination of the target time step corresponding to the first time step according to the time step interval comprises:

[0073] Determining the interval endpoint of the time step interval, and determining the target time step corresponding to the first time step according to the interval endpoint.

[0074] Continuing with the above example, if the first time step is the 20th time step, and the time step interval corresponding to the first time step is 0-50, then the target time step corresponding to the first time step is determined according to the interval endpoint of the time step interval, which is the 0th time step or the 50th time step.

[0075] Since each time step interval includes two interval endpoints, determining the target time step corresponding to the first time step according to the interval endpoints of the time step interval can include two cases. In one case, the interval endpoints of the time step interval are interval left endpoints, and the target time step corresponding to the first time step is determined according to the interval endpoints of the time step interval, which is the interval left endpoint of the time step interval. Similarly, in another case, the interval endpoints of the time step interval are interval right endpoints, and the target time step corresponding to the first time step is determined according to the interval endpoints of the time step interval, which is the interval right endpoint of the time step interval.

[0076] First, taking the interval left endpoint of the time step interval as an example, the determination of the target time step corresponding to the first time step according to the interval endpoints is described in detail as follows:

[0077] The determination of the target time step corresponding to the first time step according to the interval endpoints includes:

[0078] The interval left endpoint of the time step interval is determined, and the interval left endpoint is determined as the target time step corresponding to the first time step, wherein the interval right endpoint of the time step interval is the left endpoint included in the next time step interval.

[0079] Taking the first time step as the 53rd time step and the time step interval corresponding to the first time step as 50-100 as an example.

[0080] The interval left endpoint of the time step interval 50-100 is 50, so the 50th time step in the time step interval can be taken as the target time step corresponding to the first time step. That is, as long as the first time step belongs to the time step interval 50-100, the target time step corresponding to the first time step is the interval left endpoint of the time step interval, i.e., the 50th time step.

[0081] Continuing with the above example, if the time step set includes 1000 time steps and is divided into 20 time step intervals, the time step interval including 50-100 time steps can be represented as [50, 100), i.e., the interval right endpoint of the time step interval is the left endpoint included in the next time step interval, i.e., the time step interval including 100-150 time steps can be represented as [100, 150).

[0082] The diffusion model processing method provided in the embodiments of the present specification can take the time step corresponding to the interval left endpoint of the time step interval corresponding to each time step as the target time step corresponding to each time step. Then, when subsequent diffusion model training is performed, the target time step corresponding to the interval left endpoint of the time step interval corresponding to each time step can be taken as the time step condition, thereby reducing the time step condition during diffusion model training, improving the training efficiency and training effect of the diffusion model.

[0083] Secondly, taking the interval right endpoint of the time step interval as an example, the determination of the target time step corresponding to the first time step according to the interval endpoint is described in detail, and the specific implementation manner is as follows:

[0084] The determination of the target time step corresponding to the first time step according to the interval endpoint includes:

[0085] The interval right endpoint of the time step interval is determined, and the interval right endpoint is determined as the target time step corresponding to the first time step, wherein the interval left endpoint of the time step interval is the right endpoint included in the previous time step interval.

[0086] In the above example, still taking the first time step as the 53rd time step, and taking the time step interval of 50-100 as an example for description.

[0087] The interval right endpoint of the time step interval 50-100 is 100, so the 100th time step in the time step interval can be taken as the target time step corresponding to the first time step. That is, as long as the first time step belongs to the time step interval 50-100, the target time step corresponding to the first time step is the interval right endpoint of the time step interval, i.e., the 100th time step.

[0088] Still in the above example, if the time step set includes 1000 time steps and is divided into 20 time step intervals, the time step interval including 50-100 time steps can be represented by (50, 100], i.e., the interval left endpoint of the time step interval is the right endpoint included in the previous time step interval, i.e., the time step interval including 100-150 time steps can be represented by [50, 100).

[0089] The diffusion model processing method provided in the embodiments of the present specification can take the time step corresponding to the interval right endpoint of the time step interval corresponding to each time step as the target time step corresponding to each time step. Then, when subsequent diffusion model training is performed, the target time step corresponding to the interval left endpoint of the time step interval corresponding to each time step can be taken as the time step condition, thereby reducing the time step condition during diffusion model training, improving the training efficiency and training effect of the diffusion model.

[0090] Of course, it is also not excluded that the target time step corresponding to the first time step can be implemented according to the middle time step of the time step interval. In practical application, it can be found from an observation that different time steps t of the network input, if the values of the time steps t are relatively close, the network prediction is also very close for the same input (this is caused by the continuity of the network), so reducing the time step t can reduce the burden of the network; and the values of the time steps in the same time step interval are relatively close, so any one of the time steps in each time step interval can be selected as the target time step corresponding to the first time step.

[0091] Step 306: input the first time step corresponding to the noise-added picture and the target time step into the diffusion model to obtain the predicted noise corresponding to the noise-added picture.

[0092] The first time step corresponding to the noise-added picture can be understood as a noise-added picture that is added with noise at the first time step in the forward process of the diffusion model.

[0093] Specifically, the diffusion model is divided into two stages, including a forward process and a reverse process. The forward process is to construct a Markov chain to gradually add noise to the picture signal to become a noise signal, that is, a picture with noise. Specifically, a discrete Markov chain {x0, x1,.., xT} is first constructed, and the transition probability of the forward process can be expressed as formula 1: N} is a pre-designed noise sequence.

[0094]

[0095] wherein, {β0, β1,..., β N} is a pre-designed noise sequence.

[0096] According to formula 1, when the forward process reaches T steps, the distribution is as shown in formula 2:

[0097]

[0098] The distribution is very close to the standard normal distribution, so the reverse generation process can be directly sampled from a Gaussian distribution.

[0099] Therefore, according to the above formula 1 and formula 2, the noise-added picture corresponding to each time step in the forward process of the diffusion model can be obtained. The specific implementation mode is as follows:

[0100] Before the first time step corresponding to the noise-added picture and the target time step are input into the diffusion model to obtain the predicted noise corresponding to the noise-added picture, the method further includes:

[0101] determine an initial picture and target noise corresponding to the first time step;

[0102] determine a noised picture corresponding to the first time step and target noise corresponding to the noised picture according to the initial picture and the target noise.

[0103] The initial picture can be understood as an un-noised picture of any size and any format, or a noised picture output by a previous time step of the first time step.

[0104] Specifically, in the case of the first time step being an original picture, the initial picture and the target noise corresponding to the first time step are determined, and the target noise is added to the initial picture to generate the noised picture corresponding to the first time step and the target noise corresponding to the noised picture. In the case of the first time step being a picture other than the original picture, the initial picture can be understood as a noised picture output by a previous time step of the first time step. Then, the initial picture and the target noise corresponding to the first time step are determined, and the target noise is added to the initial picture to generate the noised picture corresponding to the first time step and the target noise corresponding to the noised picture. Of course, in the process of adding noise to the initial picture by the forward process of the diffusion model, not only the target noise corresponding to the first time step is added, but also the noise intensity corresponding to the first time step is considered. The specific process of adding noise to the picture is not limited in the embodiments of the present disclosure.

[0105] In a specific implementation, the noised picture corresponding to the first time step and the target time step corresponding to the first time step are input into the diffusion model to obtain the predicted noise corresponding to the noised picture.

[0106] Step 308: processing the diffusion model according to the target noise corresponding to the noised picture and the predicted noise.

[0107] Specifically, the processing of the diffusion model according to the target noise corresponding to the noised picture and the predicted noise includes:

[0108] calculating a noise loss function according to the target noise corresponding to the noised picture and the predicted noise, adjusting network parameters of the diffusion model according to the noise loss function, and obtaining the diffusion model when a preset training end condition is met.

[0109] In practical applications, after determining the target noise corresponding to the noisy image and the predicted noise output by the diffusion model, a noise loss function can be calculated based on the difference between the target noise and the predicted noise. Subsequently, the network parameters of the diffusion model can be adjusted according to this noise loss function to train the model. The trained diffusion model is obtained when a preset training termination condition is met. This preset training termination condition can be understood as the number of training iterations meeting a preset threshold (e.g., 10,000 or 20,000 iterations); or the model performance (e.g., accuracy, precision, etc.) meeting a preset performance threshold.

[0110] In practical implementation, the calculation process of this noise loss function can be found in the following formula 3:

[0111]

[0112] In this equation, Equation 3 represents the noise loss function used in training. During training, firstly, t is sampled, where t follows a uniform distribution from 0 to T, which is equivalent to randomly selecting an integer from 0 to T as t. Next, x0 is sampled, which is equivalent to sampling a real image from the dataset. Finally, a noise ∈ is sampled from a standard normal distribution. After sampling, the noise ∈ is added to the real image x0 to obtain the noisy image. The intensity of the added noise is determined by t, specifically the parameter α. t It was pre-designed. The image with added noise was then x... t and Input the noise into the diffusion model, and calculate the loss by comparing its output (i.e., predicted noise) with the actual noise (the goal of training is to enable the neural network to process noise from a noisy image x). t The noise intensity (determined by t) is predicted to be added to the image.

[0113] in,

[0114] T is a time step interval sequence T = {t0, t1, ..., t2} n The elements within this group are the left endpoints of the time step intervals. For example, if 1000 t values ​​are divided into 20 intervals, then T = {0, 50, 100, ..., 1000}. T (t) is a function that takes the current time step t as input and outputs the left endpoint of the time step interval to which t belongs. For example, the time step interval corresponding to t = 53 is 50-100, so f(t) T (53) = 50.

[0115] In summary, the single-step transition probability of the reverse process of this diffusion model can be found in Equation 4 below, which represents the reverse denoising process of this diffusion model:

[0116]

[0117] where Equation 3 represents the one-step transition probability of the reverse process of the diffusion model, i.e., given x t , x t-1 follows a Gaussian distribution with mean and variance , which is pre-set. The mean , β t is a series of fixed parameters indicating the strength of the noise added at different t in the forward process, and α t is a parameter calculated from β t : is a neural network, the input is the noisy picture x t , and the left endpoint f T (t) corresponding to t, and the output is the noise added to x t .

[0118] The diffusion model processing method provided by the embodiments of the present specification divides the set of time steps into time step intervals, so that the diffusion model shares the time step conditions within a time step interval during subsequent diffusion model training, that is, the first time step shares the target time step within the corresponding time step interval, which reduces the time step conditions and greatly reduces the training burden and improves the model training performance.

[0119] In addition, after the diffusion model is trained, the diffusion model can be applied in practice. The specific implementation manner is as follows:

[0120] The diffusion model processing method further includes:

[0121] determining a target noisy picture, inputting the target noisy picture into the diffusion model, and obtaining a predicted noise corresponding to the target noisy picture;

[0122] determining a denoised target picture according to the target noisy picture and the predicted noise corresponding to the target noisy picture.

[0123] The target noisy picture can be understood as a noisy picture of any size and any format.

[0124] In another implementation manner, the target noisy picture can be understood as a video frame, that is, the diffusion model can denoise the noisy video frame to obtain a clear and accurate video frame. The specific implementation manner is as follows:

[0125] The method further includes:

[0126] A set of video frames to be noise-added is determined, and any video frame in the set is identified as the target image to be noise-added.

[0127] The set of noisy video frames includes multiple video frames of any type with various noises added.

[0128] Specifically, after determining the set of video frames to be denoised, any one of the video frames in the set can be used as the target image to be denoised for subsequent denoising processing.

[0129] After determining the target noisy image, it can be directly input into the diffusion model trained using the above diffusion model processing method. The diffusion model can then output the predicted noise corresponding to the target noisy image. Then, the noise in the target noisy image can be removed based on the predicted noise, thus accurately obtaining the denoised target image.

[0130] In another possible embodiment, the diffusion model processing method provided in this specification can also be applied to the field of text-to-image (AI) generation; that is, based on the diffusion model as an image generation model, text is used as a condition for AI image generation; in specific implementation, the text condition can be encoded using a pre-trained encoder, and then the encoding is combined with the diffusion model through a self-attention mechanism. Based on the image generation by the diffusion model, the encoding is combined to generate an AI image regulated by the text condition.

[0131] For example, if the text condition is: generate an AI image of a teddy bear skateboarding in Times Square, then the text is input into the AI ​​image generation model (i.e., text encoder + diffusion model). The text encoder of the AI ​​image generation model encodes the text, and then the encoding is combined with the diffusion model through self-attention to generate the AI ​​image. Finally, the AI ​​image generation model can output an AI image of a teddy bear skateboarding in Times Square.

[0132] In practical applications, this method can generate not only AI images but also ordinary 2D images, depending on the specific application and the conditions set. This manual does not impose any limitations on this.

[0133] See Figure 4 , Figure 4 A flowchart of an image processing method provided in one embodiment of this specification is shown, which specifically includes the following steps.

[0134] Step 402: Determine the target noisy image, input the target noisy image into the diffusion model, and obtain the predicted noise corresponding to the target noisy image.

[0135] Step 404: determining a denoised target picture according to the target noisy picture and the prediction noise corresponding to the target noisy picture,

[0136] The diffusion model is obtained by using the diffusion model processing method.

[0137] Specifically, the specific implementation steps of the picture processing method can refer to the detailed description of the diffusion model processing method in the above embodiments, and the embodiments of the present specification will not be repeated here.

[0138] The picture processing method provided by the embodiments of the present specification can quickly and accurately perform picture denoising by using a diffusion model with high performance and trained under time step conditions, to obtain a denoised target picture, greatly improving the picture denoising performance.

[0139] Corresponding to the method embodiments, the present specification also provides diffusion model processing device embodiments, Figure 5 A structural schematic diagram of a diffusion model processing device provided by an embodiment of the present specification is shown. As shown in the figure, Figure 5 The device comprises:

[0140] The interval division module 502 is configured to determine a time step set of a diffusion model and time step intervals corresponding to the time step set;

[0141] The target time step determination module 504 is configured to determine a first time step from the time step set and determine a target time step corresponding to the first time step according to the time step intervals, wherein the first time step is any time step in the time step set;

[0142] The first model prediction module 506 is configured to input the first time step corresponding noisy picture and the target time step into the diffusion model to obtain the prediction noise corresponding to the noisy picture;

[0143] The model processing module 508 is configured to process the diffusion model according to the target noise corresponding to the noisy picture and the prediction noise.

[0144] Optionally, the device further comprises:

[0145] The noise adding module is configured to:

[0146] determine an initial picture and target noise corresponding to the first time step;

[0147] determine a noisy picture corresponding to the first time step and target noise corresponding to the noisy picture according to the initial picture and the target noise.

[0148] Optionally, the interval division module 502 is further configured to:

[0149] determine a time step set of the diffusion model, and divide time steps in the time step set into time step intervals according to a preset division condition, to obtain time step intervals corresponding to the time step set.

[0150] Optionally, the target time step determination module 504 is further configured to:

[0151] determine an interval endpoint of the time step interval, and determine the target time step corresponding to the first time step according to the interval endpoint.

[0152] Optionally, the target time step determination module 504 is further configured to:

[0153] determine an interval left endpoint of the time step interval, and determine the target time step corresponding to the first time step as the interval left endpoint, wherein an interval right endpoint of the time step interval is a left endpoint included in a next time step interval.

[0154] Optionally, the target time step determination module 504 is further configured to:

[0155] determine an interval right endpoint of the time step interval, and determine the target time step corresponding to the first time step as the interval right endpoint, wherein an interval left endpoint of the time step interval is a right endpoint included in a previous time step interval.

[0156] Optionally, the model processing module 508 is further configured to:

[0157] calculate a noise loss function according to the target noise corresponding to the noisy picture and the predicted noise, adjust network parameters of the diffusion model according to the noise loss function, and obtain the diffusion model in a case where a preset training end condition is met.

[0158] Optionally, the apparatus further includes:

[0159] a denoising module configured to:

[0160] determine a target noisy picture, input the target noisy picture into the diffusion model, and obtain a predicted noise corresponding to the target noisy picture;

[0161] determine a denoised target picture according to the target noisy picture and the predicted noise corresponding to the target noisy picture.

[0162] Optionally, the denoising module is further configured to:

[0163] A set of noisy video frames is determined, and any video frame in the set of noisy video frames is determined as a target noisy picture.

[0164] The diffusion model processing apparatus provided by the embodiment of the present specification divides the set of time steps into time step intervals, and enables the diffusion model to share time step conditions within a time step interval during subsequent diffusion model training, i.e., each first time step shares a target time step within the corresponding time step interval, thereby reducing the time step conditions and greatly reducing the training burden and improving the model training performance.

[0165] The above is a schematic scheme of the diffusion model processing apparatus of the embodiment. It should be noted that the technical scheme of the diffusion model processing apparatus and the technical scheme of the diffusion model processing method described above belong to the same concept, and the details of the technical scheme of the diffusion model processing apparatus that are not described in detail can be referred to the description of the technical scheme of the diffusion model processing method.

[0166] Corresponding to the method embodiment described above, the present specification also provides a picture processing apparatus embodiment, Figure 6 A structural schematic diagram of a picture processing apparatus provided by an embodiment of the present specification is shown. As shown in the figure, Figure 6 The apparatus includes:

[0167] The second model prediction module 602 is configured to determine a target noisy picture, input the target noisy picture into a diffusion model, and obtain predicted noise corresponding to the target noisy picture;

[0168] The target picture determination module 604 is configured to determine a denoised target picture according to the target noisy picture and the predicted noise corresponding to the target noisy picture,

[0169] The diffusion model is obtained by the diffusion model processing method described above.

[0170] The picture processing apparatus provided by the embodiment of the present specification can quickly and accurately perform picture denoising by using a high-performance diffusion model trained with reduced time step conditions, and obtain a denoised target picture, thereby greatly improving the picture denoising performance.

[0171] The above is a schematic scheme of the picture processing apparatus of the embodiment. It should be noted that the technical scheme of the picture processing apparatus and the technical scheme of the picture processing method described above belong to the same concept, and the details of the technical scheme of the picture processing apparatus that are not described in detail can be referred to the description of the technical scheme of the picture processing method.

[0172] Figure 7A structural block diagram of a computing device 700 is shown, according to one embodiment of the present specification. The components of the computing device 700 include, but are not limited to, a memory 710 and a processor 720. The processor 720 is connected to the memory 710 through a bus 730, and a database 750 is used to store data.

[0173] The computing device 700 also includes an access device 740 that enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 740 can include one or more of any type of network interface (e.g., a network interface card (NIC)), wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and the like.

[0174] In one embodiment of the present specification, the above-mentioned components of the computing device 700 and other components not shown in the Figure 7 may be connected to each other, for example, through a bus. It should be understood that Figure 7 The structural block diagram of the computing device shown is only for the purpose of example, and is not a limitation on the scope of the present specification. Other components can be added or replaced as needed by those skilled in the art.

[0175] The computing device 700 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other type of mobile device, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 700 can also be a mobile or stationary server.

[0176] The processor 720 is configured to execute computer-executable instructions to perform the steps of the diffusion model processing method or the picture processing method described above. The above is a schematic solution of the computing device according to an embodiment of the present disclosure. It should be noted that the technical solution of the computing device and the technical solution of the diffusion model processing method or the picture processing method described above belong to the same concept, and the details of the technical solution of the computing device not described in detail can be referred to the description of the technical solution of the diffusion model processing method or the picture processing method.

[0177] An embodiment of the present disclosure further provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the diffusion model processing method or the picture processing method described above.

[0178] The above is a schematic solution of the computer-readable storage medium according to an embodiment of the present disclosure. It should be noted that the technical solution of the storage medium and the technical solution of the diffusion model processing method or the picture processing method described above belong to the same concept, and the details of the technical solution of the storage medium not described in detail can be referred to the description of the technical solution of the diffusion model processing method or the picture processing method.

[0179] An embodiment of the present disclosure further provides a computer program, which, when executed in a computer, causes the computer to perform the steps of the diffusion model processing method or the picture processing method described above.

[0180] The above is a schematic solution of the computer program according to an embodiment of the present disclosure. It should be noted that the technical solution of the computer program and the technical solution of the diffusion model processing method or the picture processing method described above belong to the same concept, and the details of the technical solution of the computer program not described in detail can be referred to the description of the technical solution of the diffusion model processing method or the picture processing method.

[0181] The above-described embodiments of the application have several aspects, no single one of which is solely responsible for the application's desirable attributes. Without limiting the scope of the application as expressed by the claims which follow, some further embodiments make these aspects even more useful. Other embodiments can result in less desirable attributes.

[0182] The computer readable medium can include any entity or apparatus capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, software distribution medium, etc. It should be noted that the computer readable medium can include appropriate contents according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0183] It should be noted that for the foregoing method embodiments, the acts described can be performed in a different order than that described, and that various interlocking and / or parallel configurations are also possible according to the certain embodiments of the present specification. Furthermore, certain of the acts can be optional depending upon the particular embodiment of the method. The scope of the method of each method claim should not be limited to the specific embodiments set forth herein, but should be given the full scope of its corresponding jurisdictional patent claims and any equivalents thereof.

[0184] In the above embodiments, the description of each embodiment is focused on different aspects, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0185] The above disclosed preferred embodiments of the present specification are only used to help explain the present specification. The alternative embodiments do not describe all the details and do not limit the application to the specific embodiments described. Obviously, according to the content of the embodiments of the present specification, many modifications and changes can be made. The present specification selects and describes these embodiments in order to better explain the principles and practical applications of the embodiments of the present specification, so that those skilled in the art can well understand and use the present specification. The present specification is limited by the claims and their full scope and equivalents.

Claims

1. A diffusion model processing method, comprising: Determine the set of time steps for the diffusion model, and the time step intervals corresponding to the set of time steps; A first time step is determined from the set of time steps, and a target time step corresponding to the first time step is determined according to the time step interval. The first time step is any time step in the set of time steps, and each time step interval corresponds to a target time step. The target time step serves as the time step condition corresponding to the time step interval during the training of the diffusion model. The noisy image corresponding to the first time step and the target time step are input into the diffusion model to obtain the predicted noise corresponding to the noisy image. The diffusion model is processed based on the target noise corresponding to the noisy image and the predicted noise.

2. The diffusion model processing method according to claim 1, before inputting the noisy image corresponding to the first time step and the target time step into the diffusion model to obtain the predicted noise corresponding to the noisy image, further comprising: Determine the initial image and the target noise corresponding to the first time step; Based on the initial image and the target noise, determine the noisy image corresponding to the first time step and the target noise corresponding to the noisy image.

3. The diffusion model processing method according to claim 1, wherein determining the time step set of the diffusion model and the time step interval corresponding to the time step set includes: A time step set for the diffusion model is determined, and the time steps in the time step set are divided into intervals according to preset partitioning conditions to obtain the time step intervals corresponding to the time step set.

4. The diffusion model processing method according to claim 1, wherein determining the target time step corresponding to the first time step based on the time step interval includes: Determine the endpoints of the time step interval, and determine the target time step corresponding to the first time step based on the endpoints of the interval.

5. The diffusion model processing method according to claim 4, wherein determining the interval endpoints of the time step interval and determining the target time step corresponding to the first time step based on the interval endpoints includes: The left endpoint of the time step interval is determined, and the left endpoint of the interval is determined as the target time step corresponding to the first time step, wherein the right endpoint of the time step interval is the left endpoint included in the next time step interval.

6. The diffusion model processing method according to claim 4, wherein determining the interval endpoints of the time step interval and determining the time step corresponding to the first time step based on the interval endpoints includes: The right endpoint of the time step interval is determined, and the right endpoint of the interval is determined as the target time step corresponding to the first time step, wherein the left endpoint of the time step interval is the right endpoint included in the previous time step interval.

7. The diffusion model processing method according to claim 1, wherein processing the diffusion model based on the target noise corresponding to the noisy image and the predicted noise includes: The noise loss function is calculated based on the target noise corresponding to the noisy image and the predicted noise. The network parameters of the diffusion model are adjusted according to the noise loss function, and the diffusion model is obtained when the preset training termination condition is met.

8. The diffusion model processing method according to any one of claims 1-7, further comprising, after processing the diffusion model based on the target noise corresponding to the noisy image and the predicted noise: Determine the target noisy image, input the target noisy image into the diffusion model, and obtain the predicted noise corresponding to the target noisy image; The denoised target image is determined based on the target image with added noise and the predicted noise corresponding to the target image with added noise.

9. The diffusion model processing method according to claim 8, wherein determining the target noisy image includes: A set of video frames to be noise-added is determined, and any video frame in the set is identified as the target image to be noise-added.

10. A diffusion model processing apparatus, comprising: The interval partitioning module is configured to determine the set of time steps of the diffusion model and the time step intervals corresponding to the set of time steps; The target time step determination module is configured to determine a first time step from the set of time steps, and determine a target time step corresponding to the first time step according to the time step interval, wherein the first time step is any time step in the set of time steps, each time step interval corresponds to a target time step, and the target time step serves as the time step condition corresponding to the time step interval during the training of the diffusion model. The first model prediction module is configured to input the noisy image corresponding to the first time step and the target time step into the diffusion model to obtain the predicted noise corresponding to the noisy image; The model processing module is configured to process the diffusion model based on the target noise corresponding to the noisy image and the predicted noise.

11. An image processing method, comprising: Determine the target noisy image, input the target noisy image into the diffusion model, and obtain the predicted noise corresponding to the target noisy image; Based on the noisy target image and the predicted noise corresponding to the noisy target image, the denoised target image is determined. The diffusion model is obtained by the diffusion model processing method described in any one of claims 1-9.

12. An image processing apparatus, comprising: The second model prediction module is configured to determine the target noisy image, input the target noisy image into the diffusion model, and obtain the predicted noise corresponding to the target noisy image. The target image determination module is configured to determine the denoised target image based on the noisy target image and the predicted noise corresponding to the noisy target image. The diffusion model is obtained by the diffusion model processing method described in any one of claims 1-9.

13. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the diffusion model processing method according to any one of claims 1 to 9 or the image processing method according to claim 11.

14. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the diffusion model processing method of any one of claims 1 to 9 or the image processing method of claim 11.

Images