Model training method, image generation method and device

Abstract

The invention discloses a model training method and device and an image generation method and device. Belongs to the technical field of artificial intelligence. The embodiment of the model training method comprises the following steps: acquiring a sample image; on the basis of beta distribution, time steps adopted by the diffusion model in the diffusion process are sampled, sampling time steps are obtained, and parameter values of beta distribution are determined on the basis of the change trend of image pixel value distribution in the forward diffusion process of the diffusion model; and training the diffusion model based on the sample image and the sampling time step.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and specifically to a model training method, an image generation method, and a device thereof. Background Art

[0002] The Diffusion Model (DM) is a generative model used in the field of deep learning and artificial intelligence, which performs well in image generation. The workflow of the diffusion model can be divided into a forward process and a reverse process. The forward process is also called a diffusion process, which is used to add noise to the image to change the distribution of image pixel values. The reverse process is also called a denoising process, which is used to remove image noise to obtain a clear image. In the diffusion model training stage, it is usually necessary to sample the time step, determine the added Gaussian noise based on the sampled time step, and perform model training based on the noise and the noise predicted by the diffusion model.

[0003] In the prior art, in the diffusion model training process, time steps are usually uniformly sampled. However, the changes in the distribution of image pixel values ​​at different time steps are non-uniform. When the time step is small, the changes in the distribution of image pixel values ​​are more drastic, and when the time step is large, the changes in the distribution of image pixel values ​​are relatively gentle. Uniform sampling ignores the changing nature of image pixel values ​​during the diffusion process. Therefore, the diffusion model does not pay enough attention to the stage where the image pixel values ​​change greatly, resulting in low training efficiency and low image generation quality. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a model training method, an image generation method and a device thereof, which can improve the training efficiency of the diffusion model and the image generation quality.

[0005] In a first aspect, an embodiment of the present application provides a model training method, the method comprising: acquiring a sample image; based on a beta distribution, sampling the time steps used by the diffusion model in the diffusion process to obtain sampling time steps, wherein the parameter value of the beta distribution is determined based on the changing trend of the image pixel value distribution during the diffusion process; and training the diffusion model based on the sample image and the sampling time steps.

[0006] In a second aspect, an embodiment of the present application provides a model training device, which includes: an acquisition unit for acquiring a sample image; a sampling unit for sampling the time steps used by the diffusion model in the diffusion process based on a beta distribution to obtain a sampling time step, wherein the parameter value of the beta distribution is determined based on a changing trend of the image pixel value distribution during the diffusion process; and a training unit for training the diffusion model based on the sample image and the sampling time step.

[0007] In a third aspect, an embodiment of the present application provides an image generation method, the method comprising: obtaining an image to be denoised; inputting the image to be denoised into a diffusion model trained using the model training method described in the first aspect to obtain noise data output by the diffusion model; and generating a target image based on the image to be denoised and the noise data.

[0008] In a fourth aspect, an embodiment of the present application provides an image generating device, which includes: an acquisition unit, used to acquire an image to be denoised; a processing unit, used to input the image to be denoised into a diffusion model trained by the model training method described in the first aspect, to obtain noise data output by the diffusion model; and a generation unit, used to generate a target image based on the image to be denoised and the noise data.

[0009] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect or the second aspect are implemented.

[0010] In a sixth aspect, an embodiment of the present application provides a readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first aspect or the second aspect are implemented.

[0011] In the seventh aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the method described in the first aspect or the second aspect.

[0012] In an eighth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect or the second aspect.

[0013] In the embodiment of the present application, a sample image is first obtained; then, based on the beta distribution, the time step used by the diffusion model in the diffusion process is sampled to obtain the sampling time step, and the parameter value of the beta distribution is determined based on the changing trend of the distribution of the image pixel value in the diffusion process; finally, based on the sample image and the sampling time step, the diffusion model is trained. Since the parameter value of the beta distribution is determined based on the changing trend of the distribution of the image pixel value in the diffusion process, the beta distribution can have the same changing trend as the distribution of the image pixel value in the diffusion process of the diffusion model. On this basis, based on the sampling of the time step of the beta distribution, on the one hand, the time step in which the distribution of the image pixel value changes drastically during the diffusion process can be sampled more densely, so that the diffusion model can learn the characteristics of the stage in which the distribution of the image pixel value changes greatly more quickly, and at the same time, the time step in which the distribution of the image pixel value changes slowly is sampled more sparsely, so that the diffusion model can converge faster, thereby improving the training efficiency of the diffusion model; on the other hand, by sampling more densely the time step in which the distribution of the image pixel value changes drastically during the diffusion process, the diffusion model can pay more attention to the stage in which the distribution of the image pixel value changes greatly during the training process, which helps to improve the quality of the image generated by the diffusion model. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a flow chart of the model training method provided in an embodiment of the present application;

[0015] Figure 2 is a flow chart of an image generation method provided in an embodiment of the present application;

[0016] Figure 3 It is a structural schematic diagram of a model training device provided in an embodiment of the present application;

[0017] Figure 4 is a schematic diagram of the structure of an image generating device provided in an embodiment of the present application;

[0018] Figure 5 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0019] Figure 6 It is a schematic diagram of the hardware structure of an electronic device suitable for implementing the embodiments of the present application. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0021] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0022] The model training method and device provided in the embodiments of the present application are described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0023] Please refer to Figure 1 , which shows a flow chart of the model training method provided in the embodiment of the present application. The model training method provided in the embodiment of the present application can be applied to electronic devices. In practice, the above electronic devices can be electronic devices such as servers and desktop computers.

[0024] The process of the model training method provided in the embodiment of the present application includes the following steps:

[0025] Step 101: Obtain a sample image.

[0026] In this embodiment, a sample set may be provided, and the sample set may include a large number of sample images, and the sample images may be clear images. The training of the diffusion model may be iteratively performed multiple times, and during each training, any sample image may be randomly selected from the sample set.

[0027] Step 102, based on Beta distribution, sampling the time steps used by the diffusion model in the diffusion process to obtain sampling time steps, and the parameter value of Beta distribution is determined based on the change trend of the image pixel value distribution in the diffusion process.

[0028] In this embodiment, the diffusion model is a generative model used in the field of deep learning and artificial intelligence, and performs particularly well in image generation. The workflow of the diffusion model can be divided into a forward process and a reverse process. The forward process is also called a diffusion process, which is used to gradually add noise to real data until it is completely converted into noise, so that the data gradually transitions from a structured state to an unstructured state. For example, noise is gradually added to an image until it is completely converted into a noise map that conforms to a Gaussian distribution. The reverse process is also called a denoising process, which is used to start from a pure noise state, gradually remove noise, and restore to structured data, such as an image. This process is achieved by training the diffusion model, which learns how to remove noise at each step and gradually restores the characteristics of the original data.

[0029] The forward process corresponds to the training process of the diffusion model, and the reverse process corresponds to the inference process of the diffusion model. The significance of the forward process is to help the diffusion model learn the reverse process, that is, the noise obtained in the forward process can be used as a label. According to the label, the reverse process can know how the noise is added during denoising, and then conduct training. Specifically, the reverse process can predict the noise added in the forward process. Based on the above label and the predicted noise, the loss value of the diffusion model can be determined, and the parameters of the diffusion model can be updated based on the loss value.

[0030] Time step is a key concept in the diffusion model. It refers to each stage of the forward diffusion process of the diffusion model, that is, each stage that the diffusion model goes through when gradually adding noise to the image. An important feature of the diffusion process is that the noise of any time step can be sampled directly based on the original data. Therefore, there is no need to calculate the noise time by time step from the first time step each time. The noise of any time step t can be directly calculated. In view of this, for each training, only one time step needs to be sampled for a sample image.

[0031] In the related art, time steps are usually sampled uniformly, that is, the sampling probability of each time step is the same. For example, the total number of time steps is set to T, and the value range of time step t is an integer in [1, T]. If T is equal to 1000, there are 1000 time steps that can be sampled, and the probability of each time step being sampled is 1 / T. However, during the diffusion model diffusion process, the change in the distribution of image pixel values ​​at different time steps is non-uniform. Specifically, the distribution of image pixel values ​​satisfies the following formula 1:

[0032]

[0033] Among them, q t (x t |x0) means that after the initial image x0 is given, the image x corresponding to time step t t The conditional distribution of pixel values, represents Gaussian distribution, e is the natural logarithm, and I is the unit diagonal matrix.

[0034] The above formula is a form of an OU (Ornstein-Uhlenbeck) random process. Using the Fokker-Planck equation, the corresponding relationship between the image pixel value distribution and time can be established. Specifically, the diffusion model uses a diffusion process based on the OU process, and the corresponding Fokker-Planck equation is shown in the following formula 2:

[0035]

[0036] in, Represents the image x t The derivation of For the image x t The second-order derivative of represents the derivative of time step t, q t (x t ) is the image x corresponding to time step t t The pixel value distribution.

[0037] The above Fokker-Planck equation describes the forward process of the diffusion model and gives the relationship between pixel value distribution and time. When t is close to 0, the above Fokker-Planck equation is mainly dominated by the second-order term. Therefore, the Fokker-Planck equation can be simplified to a differential equation, as shown in the following formula 3:

[0038]

[0039] Therefore, the early stage of the OU process is simplified, and only the second-order term that plays a dominant role needs to be considered. For the above differential equation, the upper bound of the norm of the time derivative of the distribution change can be analyzed, see the following inequality 4:

[0040]

[0041] in, for The norm of for The norm of .

[0042] The above inequality shows that the upper bound of the rate of change of the image pixel value distribution is non-uniform and approaches infinity when the time step t is close to 0. It can be seen that in the early stage of the diffusion process, that is, when the time step is small, the change of the image pixel value distribution is more drastic.

[0043] In addition, the distribution change properties in the later stages of the diffusion process need to be considered. Based on the properties of the OU process, the upper bound of the KL (Kullback-Leibler) divergence between the distribution at different times in the diffusion process and the standard Gaussian distribution that finally converges can be obtained, see the following inequality 5:

[0044]

[0045] in, is a standard Gaussian distribution, q0(x0) is the pixel value distribution of the initial image x0, and KL represents the KL divergence, which measures the difference between the two distributions. According to the properties of the exponential function, when t is close to the late diffusion stage, the distribution changes relatively flat, and it has basically converged to the Gaussian distribution. It can be seen that in the late diffusion process, that is, when the time step is large, the distribution of image pixel values ​​changes relatively slowly.

[0046] In the related art, since the time steps are sampled in a uniform sampling manner, the changing nature of the image pixel values ​​during the diffusion process is ignored, causing the diffusion model to pay insufficient attention to the stages where the image pixel values ​​change greatly, and pay too much attention to the stages where the image pixel values ​​change greatly, resulting in low training efficiency and low image generation quality.

[0047] In this embodiment, the time step used by the diffusion model in the diffusion process can be sampled based on the Beta distribution to obtain the sampling time step. In practice, the Beta distribution is also called the Β distribution, which is a set of continuous probability distributions defined in the (0,1) interval. The probability density function f(t; α, β) of the Beta distribution is shown in the following formula 6:

[0048]

[0049] Wherein, t is the variable of the Beta distribution, which can be the normalized value of the time step in this embodiment. Γ is the gamma function, which is a special function commonly used in mathematics. The corresponding value can be directly calculated according to the given parameters. α and β are the parameters of the Beta distribution. By setting different parameter values, distributions of different shapes can be obtained.

[0050] In this embodiment, the parameter values ​​of the beta distribution, that is, the values ​​of α and β, can be determined based on the changing trend of the distribution of the image pixel values ​​during the diffusion process. Through experiments, when 0<α<1 and β=1, the beta distribution can change drastically in the early stage and tend to be gentle in the later stage, thereby aligning the changing trend of the distribution. Therefore, α can be set to a value between (0,1), for example, set to 0.8, and β can be set to 1.

[0051] By sampling the time steps used by the diffusion model in the diffusion process based on the Beta distribution, on the one hand, the time steps where the image pixel value distribution changes dramatically during the diffusion process can be sampled more densely, so that the diffusion model can learn the characteristics of the stage where the image pixel value distribution changes greatly more quickly, and at the same time, the time steps where the image pixel value distribution changes slowly are sampled more sparsely, so that the diffusion model can converge faster, thereby improving the training efficiency of the diffusion model; on the other hand, by sampling the time steps where the image pixel value distribution changes dramatically during the diffusion process more densely, the diffusion model can pay more attention to the stage where the image pixel value distribution changes greatly during the training process, which helps to improve the quality of images generated by the diffusion model.

[0052] In some optional implementations of this embodiment, based on the beta distribution, the time step used by the diffusion model in the diffusion process can be sampled according to the following steps to obtain the sampling time step:

[0053] The first step is to obtain a first set, which includes multiple candidate time steps applied to the diffusion process of the diffusion model. For example, the total number of candidate time steps is set to T, and the value range of the candidate time step t is an integer in [1, T]. If T is equal to 1000, there are 1000 candidate time steps in total.

[0054] The second step is to determine the sampling probability of each candidate time step in the first set based on the Beta distribution. Here, for each candidate time step, the candidate time step can be normalized and input into the probability density function corresponding to the Beta distribution, and the function value of the probability density function is determined as the sampling probability of the candidate time step.

[0055] The third step is to sample the candidate time steps in the first set based on the sampling probability to obtain the sampling time steps.

[0056] By determining the sampling probability of each candidate time step based on the Beta distribution and sampling the candidate time steps based on the sampling probability, the sampling of time steps based on the Beta distribution can be achieved, that is, more dense sampling is performed on the time steps where the image pixel value distribution changes dramatically during the diffusion process, and more sparse sampling is performed on the time steps where the image pixel value distribution changes slowly, thereby improving the training efficiency of the diffusion model and the image generation quality.

[0057] In some optional implementations of this embodiment, in the second step above, the following steps may be performed:

[0058] First, each candidate time step in the first set is normalized to obtain a second set, which includes the normalized values ​​corresponding to each candidate time step in the first set. For example, the total number of candidate time steps is set to T, and for a candidate time step t, t / T can be used as the normalized value corresponding to the time step.

[0059] Afterwards, each normalized value in the second set is input into the probability density function corresponding to the Beta distribution to obtain the sampling probability corresponding to each normalized value in the second set.

[0060] Finally, based on the sampling probabilities corresponding to the normalized values ​​in the second set, the sampling probabilities of the candidate time steps in the first set are determined. Here, for each normalized value in the second set, the sampling probability corresponding to the normalized value can be used as the sampling probability of the candidate time step corresponding to the normalized value.

[0061] By normalizing each candidate time step, it is possible to adapt to the value range of the variable of the Beta distribution, so that the sampling of the time step can be achieved based on the Beta distribution.

[0062] Step 103: training the diffusion model based on the sample image and the sampling time step.

[0063] In this embodiment, the sample image and the sampling time step can be input into the diffusion model, and the diffusion model can be trained using a machine learning method. It should be noted that the above steps 101 to 103 can be performed multiple times until the end of the training condition is met. The above end of the training condition can be set as needed, for example, it can include but is not limited to at least one of the following: the parameter value in the diffusion model tends to be stable and unchanged, the number of training times is greater than or equal to a threshold, etc.

[0064] In some optional implementations of this embodiment, based on the sample image and the sampling time step, the diffusion model may be trained in the following steps:

[0065] The first step is to obtain the first noise data based on the sampling time step.

[0066] Here, the first noise data may be a numerical matrix of the same size as the sample image, and each value in the matrix is ​​a noise value. The first noise data conforms to a Gaussian distribution. In practice, a schedule may be pre-set, and the schedule may include but is not limited to a linear schedule, a square schedule, a cosine schedule, and the like. The schedule may characterize the correspondence between the time step and the noise level. Based on the schedule, the noise level corresponding to the sampling time step may be determined, and the noise of the noise level may be sampled based on the Gaussian distribution to obtain the first noise data. The first noise data is predicted noise data and may be used as a label.

[0067] The second step is to add noise to the sample image based on the first noise data to obtain a noisy image. Here, the process of adding noise to the sample image is the process of adding the numerical matrix corresponding to the first noise data to the pixel value matrix corresponding to the sample image.

[0068] The third step is to input the noisy image and the sampling time step into the diffusion model, and predict the noise in the noisy image through the diffusion model to obtain the second noise data. The second noise data is the noise data predicted by the diffusion model, that is, the noise predicted by the diffusion model to act on the sample image.

[0069] The fourth step is to determine the loss value of the diffusion model based on the first noise data and the second noise data. Here, the first noise data and the second noise data can be input into the loss function of the diffusion model to obtain the loss value of the diffusion model. The loss function is a non-negative real-valued function that can be used to characterize the difference between the detection result and the true result. In general, the smaller the loss value, the better the robustness of the model. The loss function can be set according to actual needs. The loss function can refer to the following formula 7:

[0070]

[0071] Where t is the time step sampled based on Beta distribution Beta(α,β), x0 represents the sample image sampled from the sample image distribution q(x0), and ∈ represents the sample image sampled from the standard Gaussian distribution. The first noise data sampled, α t , σ t They represent the parameter values ​​in the pre-designed timetable, which can characterize the noise intensity. It means to find the mathematical expectation, ∈ θ represents the second noise data predicted by the diffusion model.

[0072] The fifth step is to update the parameters of the diffusion model based on the loss value. Here, the back propagation algorithm can be used to obtain the gradient of the loss value relative to the model parameters, and then the gradient descent algorithm is used to update the parameters of the diffusion model based on the gradient.

[0073] The above process corresponds to a training process of the diffusion model. By iteratively executing steps 101 to 103, the diffusion model can be iteratively trained. Since the sampling time steps input to the diffusion model during the training process are obtained based on the Beta distribution, the time steps in which the image pixel value distribution changes dramatically during the diffusion process can be sampled more densely, so that the number of noisy images with large changes in pixel value distribution is large, so that the diffusion model can learn the characteristics of the stage in which the image pixel value distribution changes greatly more quickly, thereby improving the training efficiency of the diffusion model and the image generation quality. In addition, since the time steps are sampled based on the Beta distribution each time during the training process, the time steps in which the image pixel value distribution changes slowly can be sampled more sparsely, so as to reduce the number of noisy images with slow changes in pixel value distribution, so that the diffusion model can converge faster, thereby further improving the training efficiency of the diffusion model.

[0074] The method provided by the above embodiment of the present application first obtains a sample image; then based on the beta distribution, the time step used by the diffusion model in the diffusion process is sampled to obtain the sampling time step, and the parameter value of the beta distribution is determined based on the change trend of the image pixel value distribution in the diffusion process; finally, based on the sample image and the sampling time step, the diffusion model is trained. Since the parameter value of the beta distribution is determined based on the change trend of the image pixel value distribution in the diffusion process, the beta distribution can have the same change trend as the image pixel value distribution in the diffusion process of the diffusion model. On this basis, based on the sampling of the time step of the beta distribution, on the one hand, the time step in which the image pixel value distribution changes drastically during the diffusion process can be sampled more densely, so that the diffusion model can learn the characteristics of the stage in which the image pixel value distribution changes more quickly, and at the same time, the time step in which the image pixel value distribution changes slowly is sampled more sparsely, so that the diffusion model can converge faster, thereby improving the training efficiency of the diffusion model; on the other hand, by sampling more densely the time step in which the image pixel value distribution changes drastically during the diffusion process, the diffusion model can pay more attention to the stage in which the image pixel value distribution changes more greatly during the training process, which helps to improve the quality of the image generated by the diffusion model.

[0075] Further references Figure 2 , which shows a flow chart of the image generation method provided in the embodiment of the present application. The flow chart of the image generation method provided in the embodiment of the present application includes the following steps:

[0076] Step 201: Obtain an image to be denoised.

[0077] In this embodiment, the image to be denoised may be any image with noise, for example, may be a randomly sampled pure noise image.

[0078] Step 202: input the image to be denoised into a diffusion model, and denoise the image to be denoised by the diffusion model to obtain a target image.

[0079] In this embodiment, the diffusion model can be obtained by training the model training method in the above embodiment, which will not be described in detail here. The image to be denoised can be input into the diffusion model, and the diffusion model gradually denoises the image to be denoised through multiple denoising processes to obtain a target image. The target image can be the image after the noise is removed, that is, a clear image.

[0080] In practice, when the diffusion model performs the i-th denoising operation, it can first predict the noise data based on the image obtained after the i-1th denoising and the time step corresponding to the i-th denoising. Then, based on the noise data, the image obtained after the i-1th denoising is further denoised to obtain the i-th denoised image. i is a positive integer.

[0081] It should be noted that the image generation method of this embodiment can be used to test the diffusion model trained in the above embodiment, and then the diffusion model can be continuously optimized according to the test results. The image generation method of this embodiment can also be a practical application method of the diffusion model trained in the above embodiment. Using the diffusion model trained in the above embodiment, the denoised image can be denoised to generate a clear image. Since the above diffusion model is based on the time step sampling of the beta distribution during the training process, it can pay more attention to and learn the characteristics of the stage where the image pixel value distribution changes greatly during the training process, thereby improving the quality of the target image generated by the diffusion model.

[0082] It should be noted that the model training method provided in the embodiment of the present application can be executed by a model training device. In the embodiment of the present application, the model training device executing the model training method is taken as an example to illustrate the model training device provided in the embodiment of the present application.

[0083] like Figure 3 As shown, the model training device 300 of this embodiment includes: an acquisition unit 301, used to acquire a sample image; a sampling unit 302, used to sample the time steps used by the diffusion model in the diffusion process based on Beta distribution to obtain sampling time steps, and the parameter value of the Beta distribution is determined based on the changing trend of the image pixel value distribution in the diffusion process; a training unit 303, used to train the diffusion model based on the sample image and the sampling time step.

[0084] In some optional implementations of this embodiment, the sampling unit 302 is further used to: obtain a first set, the first set including multiple candidate time steps applied to the diffusion process of the diffusion model; determine the sampling probability of each candidate time step in the first set based on the Beta distribution; and sample the candidate time steps in the first set based on the sampling probability to obtain the sampling time steps. By determining the sampling probability of each candidate time step based on the Beta distribution and sampling the candidate time steps based on the sampling probability, sampling of time steps based on the Beta distribution can be achieved, that is, more dense sampling is performed on time steps where the image pixel value distribution changes dramatically during the diffusion process, and more sparse sampling is performed on time steps where the image pixel value distribution changes slowly, thereby improving the training efficiency of the diffusion model and the image generation quality.

[0085] In some optional implementations of the present embodiment, the sampling unit 302 is further used to: normalize each candidate time step in the first set to obtain a second set, wherein the second set includes normalized values ​​corresponding to each candidate time step in the first set; input each normalized value in the second set into a probability density function corresponding to a Beta distribution to obtain a sampling probability corresponding to each normalized value in the second set; and determine a sampling probability of each candidate time step in the first set based on the sampling probability corresponding to each normalized value in the second set.

[0086] In some optional implementations of this embodiment, the training unit 303 is further used to: obtain first noise data based on the sampling time step; add noise to the sample image based on the first noise data to obtain a noisy image; input the noisy image and the sampling time step into the diffusion model, and predict the noise in the noisy image through the diffusion model to obtain second noise data; determine the loss value of the diffusion model based on the first noise data and the second noise data; and update the parameters of the diffusion model based on the loss value. By normalizing each candidate time step, it can adapt to the value range of the variable of the Beta distribution, so that the sampling of time steps can be achieved based on the Beta distribution.

[0087] The device provided by the above embodiment of the present application first obtains a sample image; then based on the beta distribution, the time step used by the diffusion model in the diffusion process is sampled to obtain the sampling time step, and the parameter value of the beta distribution is determined based on the change trend of the image pixel value distribution in the diffusion process; finally, the diffusion model is trained based on the sample image and the sampling time step. Since the parameter value of the beta distribution is determined based on the change trend of the image pixel value distribution in the diffusion process, the beta distribution can have the same change trend as the image pixel value distribution in the diffusion process of the diffusion model. On this basis, based on the sampling of the time step of the beta distribution, on the one hand, the time step in which the image pixel value distribution changes drastically during the diffusion process can be sampled more densely, so that the diffusion model can learn the characteristics of the stage in which the image pixel value distribution changes more quickly, and at the same time, the time step in which the image pixel value distribution changes slowly is sampled more sparsely, so that the diffusion model can converge faster, thereby improving the training efficiency of the diffusion model; on the other hand, by sampling more densely the time step in which the image pixel value distribution changes drastically during the diffusion process, the diffusion model can pay more attention to the stage in which the image pixel value distribution changes more during the training process, which helps to improve the quality of the image generated by the diffusion model.

[0088] It should be noted that the image generation method provided in the embodiment of the present application can be executed by an image generation device. In the embodiment of the present application, the image generation device executing the image generation method is taken as an example to illustrate the image generation device provided in the embodiment of the present application.

[0089] like Figure 4 As shown, the image generating device 400 of this embodiment includes: an acquisition unit 401, used to acquire the image to be denoised; a processing unit 402, used to input the image to be denoised into a diffusion model, denoise the image to be denoised through the diffusion model, and obtain a target image.

[0090] It is understandable that the units recorded in the device 400 correspond to the steps in the above-mentioned image generation method. Therefore, the operations, features and beneficial effects described above for the image generation method are also applicable to the device 400 and the units contained therein, and will not be repeated here.

[0091] The model training device and image generation device in the embodiment of the present application can be an electronic device, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplary, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (Augmented Reality, AR) / virtual reality (Virtual Reality, VR) device, a robot, a wearable device, a super mobile personal computer (Ultra-Mobile Personal Computer, UMPC), a netbook or a personal digital assistant (Personal Digital Assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (Personal Computer, PC), a television (Television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0092] The model training device and the image generation device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0093] The model training device provided in the embodiment of the present application can achieve Figure 1 The various processes implemented by the method embodiment are not described here in detail to avoid repetition. The image generation device provided in the embodiment of the present application can achieve Figure 2 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0094] Alternatively, if Figure 5 As shown, an embodiment of the present application also provides an electronic device 500, including a processor 501 and a memory 502, wherein the memory 502 stores programs or instructions that can be executed on the processor 501, and when the program or instructions are executed by the processor 501, the various steps of the above-mentioned model training method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0095] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0096] Figure 6 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0097] The electronic device 600 includes but is not limited to: a radio frequency unit 601, a network module 602, an audio output unit 603, an input unit 604, a sensor 605, a display unit 606, a user input unit 607, an interface unit 608, a memory 609, and a processor 610.

[0098] Those skilled in the art will appreciate that the electronic device 600 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 610 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 6 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0099] Among them, the processor 610 is used to obtain a sample image; based on the Beta distribution, the time step used by the diffusion model in the diffusion process is sampled to obtain a sampling time step, and the parameter value of the Beta distribution is determined based on the changing trend of the image pixel value distribution during the diffusion process; based on the sample image and the sampling time step, the diffusion model is trained.

[0100] Since the parameter value of the Beta distribution is determined based on the changing trend of the distribution of image pixel values ​​during the diffusion process, the Beta distribution can have the same changing trend as the distribution of image pixel values ​​during the diffusion process of the diffusion model. On this basis, based on the sampling of time steps based on the Beta distribution, on the one hand, the time steps where the distribution of image pixel values ​​changes dramatically during the diffusion process can be sampled more densely, so that the diffusion model can learn the characteristics of the stage where the distribution of image pixel values ​​changes greatly more quickly, and at the same time, the time steps where the distribution of image pixel values ​​changes slowly can be sampled more sparsely, so that the diffusion model can converge faster, thereby improving the training efficiency of the diffusion model; on the other hand, by sampling more densely the time steps where the distribution of image pixel values ​​changes dramatically during the diffusion process, the diffusion model can pay more attention to the stage where the distribution of image pixel values ​​changes greatly during the training process, which helps to improve the quality of the images generated by the diffusion model.

[0101] Optionally, the processor 610 is further configured to obtain a first set, wherein the first set includes a plurality of candidate time steps applied to a diffusion process of a diffusion model; determine a sampling probability of each candidate time step in the first set based on a Beta distribution; and sample the candidate time steps in the first set based on the sampling probability to obtain a sampling time step. By determining the sampling probability of each candidate time step based on a Beta distribution and sampling the candidate time steps based on the sampling probability, sampling of time steps based on a Beta distribution can be achieved, that is, more dense sampling is performed on time steps where the distribution of image pixel values ​​changes dramatically during the diffusion process, and more sparse sampling is performed on time steps where the distribution of image pixel values ​​changes slowly, thereby improving the training efficiency of the diffusion model and the image generation quality.

[0102] Optionally, the processor 610 is further used to normalize each candidate time step in the first set to obtain a second set, wherein the second set includes normalized values ​​corresponding to each candidate time step in the first set; input each normalized value in the second set to a probability density function corresponding to a Beta distribution to obtain a sampling probability corresponding to each normalized value in the second set; and determine a sampling probability of each candidate time step in the first set based on the sampling probability corresponding to each normalized value in the second set.

[0103] Optionally, the processor 610 is further configured to obtain first noise data based on the sampling time step; add noise to the sample image based on the first noise data to obtain a noisy image; input the noisy image and the sampling time step into the diffusion model, and predict the noise in the noisy image through the diffusion model to obtain second noise data; determine a loss value of the diffusion model based on the first noise data and the second noise data; and update the parameters of the diffusion model based on the loss value. By normalizing each candidate time step, it is possible to adapt to the value range of the variable of the Beta distribution, so that the sampling of time steps can be achieved based on the Beta distribution.

[0104] In addition, the processor 610 can also be used to obtain an image to be denoised, input the image to be denoised into a diffusion model, and denoise the image to be denoised by the diffusion model to obtain a target image. Since the diffusion model is based on time step sampling of the Beta distribution during the training process, it can pay more attention to and learn the characteristics of the stage where the distribution of image pixel values ​​changes greatly during the training process, thereby improving the quality of the target image generated by the diffusion model.

[0105] It should be understood that in the embodiment of the present application, the input unit 604 may include a graphics processor (GPU) 6041 and a microphone 6042, and the graphics processor 6041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 606 may include a display panel 6061, and the display panel 6061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 607 includes a touch panel 6071 and at least one of other input devices 6072. The touch panel 6071 is also called a touch screen. The touch panel 6071 may include two parts: a touch detection device and a touch controller. Other input devices 6072 may include, but are not limited to, a physical keyboard, function keys (such as a volume control button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0106] The memory 609 can be used to store software programs and various data. The memory 609 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 609 may include a volatile memory or a non-volatile memory, or the memory 609 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 609 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0107] The processor 610 may include one or more processing units; optionally, the processor 610 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 610.

[0108] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned model training method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0109] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0110] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned model training method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0111] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0112] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned model training method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0113] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0114] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0115] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Get a sample image; Based on Beta distribution, sampling the time steps used by the diffusion model in the diffusion process to obtain sampling time steps, wherein the parameter value of the Beta distribution is determined based on the change trend of the distribution of the image pixel values ​​in the diffusion process; The diffusion model is trained based on the sample images and the sampling time steps.

2. The method according to claim 1, characterized in that: Based on the Beta distribution, the time step used by the diffusion model in the diffusion process is sampled to obtain the sampling time step, including: Obtaining a first set, wherein the first set includes a plurality of candidate time steps applied to a diffusion process of a diffusion model; Determine, based on Beta distribution, a sampling probability of each candidate time step in the first set; Based on the sampling probability, the candidate time steps in the first set are sampled to obtain sampling time steps.

3. The method according to claim 2, characterized in that The determining, based on the Beta distribution, the sampling probability of each candidate time step in the first set comprises: Normalizing each candidate time step in the first set to obtain a second set, wherein the second set includes normalized values ​​corresponding to each candidate time step in the first set; Inputting each normalized value in the second set into a probability density function corresponding to a beta distribution to obtain a sampling probability corresponding to each normalized value in the second set; Based on the sampling probabilities corresponding to the normalized values ​​in the second set, the sampling probabilities of the candidate time steps in the first set are determined.

4. The method according to claim 1, characterized in that The step of training the diffusion model based on the sample image and the sampling time step includes: Based on the sampling time step, obtaining first noise data; Based on the first noise data, adding noise to the sample image to obtain a noisy image; Inputting the noisy image and the sampling time step into the diffusion model, and predicting the noise in the noisy image by using the diffusion model to obtain second noise data; determining a loss value of the diffusion model based on the first noise data and the second noise data; Based on the loss value, parameters of the diffusion model are updated.

5. An image generation method, characterized in that: The method comprises: Obtain an image to be denoised; The image to be denoised is input into a diffusion model trained by the model training method according to any one of claims 1 to 4, and the image to be denoised is denoised by the diffusion model to obtain a target image.

6. A model training device, characterized in that: The device comprises: An acquisition unit, used for acquiring a sample image; A sampling unit, used for sampling the time step adopted by the diffusion model in the diffusion process based on Beta distribution to obtain a sampling time step, wherein the parameter value of the Beta distribution is determined based on the change trend of the distribution of the image pixel values ​​in the diffusion process; A training unit is used to train the diffusion model based on the sample image and the sampling time step.

7. The device according to claim 6, characterized in that The sampling unit is further used for: Obtaining a first set, wherein the first set includes a plurality of candidate time steps applied to a diffusion process of a diffusion model; Determine, based on Beta distribution, a sampling probability of each candidate time step in the first set; Based on the sampling probability, the candidate time steps in the first set are sampled to obtain sampling time steps.

8. The device according to claim 7, characterized in that The sampling unit is further used for: Normalizing each candidate time step in the first set to obtain a second set, wherein the second set includes normalized values ​​corresponding to each candidate time step in the first set; Inputting each normalized value in the second set into a probability density function corresponding to a beta distribution to obtain a sampling probability corresponding to each normalized value in the second set; Based on the sampling probabilities corresponding to the normalized values ​​in the second set, the sampling probabilities of the candidate time steps in the first set are determined.

9. The device according to claim 6, characterized in that The training unit is further used for: Based on the sampling time step, obtaining first noise data; Based on the first noise data, adding noise to the sample image to obtain a noisy image; Inputting the noisy image and the sampling time step into the diffusion model, and predicting the noise in the noisy image by using the diffusion model to obtain second noise data; determining a loss value of the diffusion model based on the first noise data and the second noise data; Based on the loss value, parameters of the diffusion model are updated.

10. An image generating device, characterized in that: The device comprises: An acquisition unit, used for acquiring an image to be denoised; A processing unit is used to input the image to be denoised into a diffusion model trained by the model training method according to any one of claims 1 to 4, and denoise the image to be denoised by using the diffusion model to obtain a target image.

Images