Image generation method based on diffusion probability model, electronic equipment and storage medium

Abstract

The invention relates to an image generation method based on a diffusion probability model, electronic equipment and a storage medium, an image is generated by using a pre-trained diffusion probability model, and the training process of the diffusion probability model comprises the following steps: obtaining an original training sample, and extracting by using an image feature extraction network to obtain an image feature vector; based on the image feature vector, utilizing a softmax layer with temperature to obtain a category vector; based on the noise level of random sampling, performing noise addition on the original training sample to obtain a noise-added training sample; based on the noise-added training sample, the noise level and the category vector, obtaining a prediction sample by using a denoising neural network; and on the basis of the original training sample and the prediction sample, by calculating a training error, updating the image feature extraction network and the denoising neural network, and realizing the training of the diffusion probability model. Compared with the prior art, the method has the advantages that the generated picture is better matched with the characteristics of a specific category, and the like.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an image generation method, electronic equipment, and storage medium based on a diffusion probability model. Background Art

[0002] As a cutting-edge generative model, the diffusion probability model has shown great potential in the field of data generation in recent years. It can learn the distribution characteristics of data from a given dataset and generate new data that matches the data distribution. The training process of the diffusion probability model usually involves a process of gradually adding noise to the data until it becomes almost impossible to extract useful information from the data, and then learning a reverse process, that is, gradually recovering the original data from the noise. This unique training method enables the diffusion probability model to capture the complex structure and high-order statistical characteristics in the data.

[0003] However, although the diffusion probability model has a strong generation capability in theory, its generation effect still has certain shortcomings in practical applications. Especially in the field of image generation, the generated images often have problems such as object deformation, missing key parts of objects, and incorrect generation of key features. These problems seriously affect the quality and practicality of the generated images.

[0004] In order to improve the generation effect of the diffusion probability model, existing methods usually increase the network depth and expand the network size. However, although this method can improve the generation quality to a certain extent, it also brings great computational complexity. Traditional computing resources can no longer meet the training requirements of such large-scale networks, resulting in high deployment and inference costs of the model in practical applications. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide an image generation method, electronic device, and storage medium based on a diffusion probability model to solve or partially solve the problems of object deformation and missing key parts in generating images in the existing methods.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] In one aspect of the present invention, a method for generating an image based on a diffusion probability model is provided, wherein an image is generated using a pre-trained diffusion probability model, wherein the training process of the diffusion probability model comprises the following steps:

[0008] Obtain the original training samples and use the image feature extraction network to extract the image feature vector;

[0009] Based on the image feature vector, a category vector is obtained using a softmax layer with temperature;

[0010] Based on the noise level of random sampling, the original training sample is denoised to obtain a noisy training sample;

[0011] Based on the noisy training sample, the noise level and the category vector, a prediction sample is obtained using a denoising neural network;

[0012] Based on the original training samples and the predicted samples, the image feature extraction network and the denoising neural network are updated by calculating the training error, so as to realize the training of the diffusion probability model.

[0013] As a preferred technical solution, the image feature vector is calculated using the following formula:

[0014] F = FeatureExactor(x0; W2)

[0015] Among them, F is the image feature vector, FeatureExactor() represents the image feature extraction network, x0 represents the input original training sample, and W2 represents the weight of the trainable image feature extraction network.

[0016] As a preferred technical solution, the category vector is calculated using the following formula:

[0017]

[0018] Among them, C i is the i-th element of the category vector C, F i represents the i-th element of the image feature vector F, T is the preset temperature, and K is the preset number of categories.

[0019] As a preferred technical solution, the noise-added training sample is obtained using the following formula:

[0020]

[0021] Among them, x t is a noisy training sample, x0 is the original training sample, t is the noise level randomly sampled in the interval, and z represents the randomly sampled standard multidimensional Gaussian noise of the same dimension as the original training sample.

[0022] As a preferred technical solution, the prediction sample is obtained using the following formula:

[0023]

[0024] in, is the predicted sample, D() represents the denoising neural network, x t, t, C, and W1 represent the noisy training samples, the noise level randomly sampled in the interval, the category vector, and the trainable denoising neural network weights, respectively.

[0025] As a preferred technical solution, the training error is calculated using the following formula:

[0026]

[0027] Among them, L is the training error, MSE() is the mean square error function, represents the predicted sample, and x0 represents the original training sample.

[0028] As a preferred technical solution, the process of generating the original training samples includes the following steps:

[0029] Obtain image data, process the image to a preset size by resizing, and construct training samples.

[0030] As a preferred technical solution, the training process of the diffusion probability model also includes the following steps:

[0031] Repeat the training several times until the preset training limit is reached.

[0032] As a preferred technical solution, the process of generating an image using a pre-trained diffusion probability model includes the following steps:

[0033] A one-hot vector of a size equal to the preset number of categories is randomly sampled, and in a multi-step denoising process, the one-hot vector is input into the denoising neural network in each step to achieve image generation.

[0034] Preferably, the multi-step denoising process is iterated as follows:

[0035]

[0036] Where T is the preset number of iterations, i = T, T-1, ..., 1, D() represents the denoising neural network, G represents a one-hot vector with a size equal to the preset number of categories, W1 is the weight of the denoising neural network, and the initial y T is a standard multi-dimensional Gaussian noise sampled randomly. After T steps of iteration, y0 is the final generated image.

[0037] Another aspect of the present invention provides an electronic device, comprising: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the aforementioned image generation method based on the diffusion probability model.

[0038] Another aspect of the present invention provides a computer-readable storage medium, comprising one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the aforementioned image generation method based on the diffusion probability model.

[0039] Compared with the prior art, the present invention has at least the following beneficial effects:

[0040] The generated images are more closely matched to the characteristics of specific categories: The present invention introduces trainable category information during the training process of the diffusion probability model. Through the guidance of the category vector, the generated images are more consistent with the characteristics of specific categories, effectively solving the problems of object deformation and missing key parts in the generated images of the existing methods. At the same time, the image data set itself is not required to have category information, which reduces the difficulty of data acquisition. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A schematic diagram of a flow chart of an image generation method based on a diffusion probability model in an embodiment;

[0042] Figure 2 Schematic diagram of an electronic device in an embodiment. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0044] Example 1

[0045] In view of the problems existing in the above-mentioned prior art, this embodiment provides an image generation method based on a diffusion probability model to effectively improve the generation effect of the diffusion probability model. Taking image training as an example, the method includes the following steps:

[0046] Step S1, obtaining an image training set.

[0047] Step S2, training of the diffusion probability model. Specifically, step S2 may include steps S201-S208:

[0048] Step S201, set the denoising neural network structure in the diffusion probability model and initialize the weight W1, set the number of categories K in the training set, set the auxiliary image feature extraction network structure and initialize the weight W2, set the upper limit of the number of training times N, and set the temperature T.

[0049] The denoising neural network can use any network structure commonly used in the diffusion probability model, and the image feature extraction network uses a convolutional neural network. The layers of the convolutional neural network and the number of feature maps in each layer must be pre-set according to the scale of the problem to ensure that the final feature vector size is the pre-set number of categories K.

[0050] Preferably, for the ImageNet dataset, the number of categories K can be set to 100 and the temperature T can be set to 0.1.

[0051] Step S202: image data is obtained from the image training set for preprocessing to form training samples. The images are processed into a preset size by resizing and cropping.

[0052] Step S203: input the training sample into the image feature extraction network to form an image feature vector F.

[0053] F = FeatureExactor(x0; W2)

[0054] In the formula, x0 represents the training sample, W2 represents the weight of the image feature extraction network, and FeatureExactor represents the function corresponding to the image feature extraction network, which is determined by the specific structure of the network.

[0055] Step S204, input the image feature vector F into the softmax layer with temperature to obtain the category vector C:

[0056]

[0057] In the formula, C i represents the i-th element of C, F i represents the ith element of F, T is a preset temperature, and K is a preset number of categories. Preferably, the temperature T takes a value less than 1, so that the size difference of different elements in the finally obtained category vector C is more significant.

[0058] Step S205, randomly sampling from [0, 1] to obtain the noise level t.

[0059] For the training sample, a standard multidimensional Gaussian noise with the same dimension is randomly sampled, and the training sample is processed with noise level t and noise to form a noisy sample:

[0060]

[0061] In the formula, x0 is the training sample, z represents the noise, and x t represents the noisy training samples.

[0062] Step S206, inputting the noisy training sample, the noise level, and the category vector into the denoising neural network to obtain a prediction sample;

[0063]

[0064] In the formula, x t Represents the noisy training sample, t represents the noise level, C represents the category vector, W1 represents the weight of the denoising neural network, and D represents the function corresponding to the denoising neural network, which is determined by the specific structure of the network.

[0065] Step S207: Calculate the training error based on the prediction sample and the training sample

[0066]

[0067] In the formula, represents the predicted sample, x0 represents the training sample, and MSE represents the mean square error function.

[0068] Using the training error, the weights W1 of the denoising neural network and the weights W2 of the image feature extraction network are modified through back propagation and an appropriate optimization algorithm (such as stochastic gradient descent).

[0069] Step S208, determining whether the model training is completed, if so, saving the denoising neural network model and executing step S3, if not, returning to step S202;

[0070] Step S3, use the trained denoising neural network to generate the image. The generation process is the same as the existing diffusion probability model, the only difference is that before the generation starts, a one-hot vector of size K is pre-sampled, and then in the multi-step denoising process, this vector is also input to the denoising neural network at each step.

[0071] Specifically, the multi-step denoising process is iterated as follows:

[0072]

[0073] Where T is the preset number of iterations, i = T, T-1, ..., 1, D() represents the denoising neural network, G represents a one-hot vector with a size equal to the preset number of categories, W1 is the weight of the denoising neural network, and the initial y T is a standard multi-dimensional Gaussian noise sampled randomly. After T steps of iteration, y0 is the final generated image.

[0074] Example 2

[0075] This embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the aforementioned image generation method based on the diffusion probability model.

[0076] like Figure 2 As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The image generation method based on the diffusion probability model. Of course, in addition to the software implementation, the present invention does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0077] Example 3

[0078] This embodiment provides a computer-readable storage medium, including one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the image generation method based on the diffusion probability model as described in Example 1.

[0079] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0080] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. An image generation method based on a diffusion probability model, characterized in that: Generate an image using a pre-trained diffusion probability model, wherein the training process of the diffusion probability model includes the following steps: Obtain the original training samples and use the image feature extraction network to extract the image feature vector; Based on the image feature vector, a category vector is obtained using a softmax layer with temperature; Based on the noise level of random sampling, the original training sample is denoised to obtain a noisy training sample; Based on the noisy training sample, the noise level and the category vector, a prediction sample is obtained using a denoising neural network; Based on the original training samples and the predicted samples, the image feature extraction network and the denoising neural network are updated by calculating the training error, so as to realize the training of the diffusion probability model.

2. The image generation method based on the diffusion probability model according to claim 1, characterized in that: The image feature vector is calculated using the following formula: F = FeatureExactor(x0; W2) Among them, F is the image feature vector, FeatureExactor() represents the image feature extraction network, x0 represents the input original training sample, and W2 represents the weight of the trainable image feature extraction network.

3. The image generation method based on the diffusion probability model according to claim 1, characterized in that: The category vector is calculated using the following formula: Among them, C i is the i-th element of the category vector C, F i represents the i-th element of the image feature vector F, T is the preset temperature, and K is the preset number of categories.

4. The image generation method based on the diffusion probability model according to claim 1, characterized in that: The noise-added training sample is obtained using the following formula: Among them, x t is a noisy training sample, x0 is the original training sample, t is the noise level randomly sampled in the interval, and z represents the randomly sampled standard multidimensional Gaussian noise of the same dimension as the original training sample.

5. The image generation method based on the diffusion probability model according to claim 1, characterized in that: The prediction sample is obtained using the following formula: in, is the predicted sample, D() represents the denoising neural network, x t , t, C, and W1 represent the noisy training samples, the noise level randomly sampled in the interval, the category vector, and the trainable denoising neural network weights, respectively.

6. The image generation method based on the diffusion probability model according to claim 1, characterized in that: The training error is calculated using the following formula: Among them, L is the training error, MSE() is the mean square error function, represents the predicted sample, and x0 represents the original training sample.

7. The image generation method based on the diffusion probability model according to claim 1, characterized in that: The generation process of the original training samples includes the following steps: Obtain image data, process the image to a preset size by resizing, and construct training samples.

8. The image generation method based on the diffusion probability model according to claim 1, characterized in that: The process of generating an image using a pre-trained diffusion probability model comprises the following steps: A one-hot vector of a size equal to the preset number of categories is randomly sampled, and in a multi-step denoising process, the one-hot vector is input into the denoising neural network in each step to achieve image generation.

9. An electronic device, characterized in that: include: One or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the image generation method based on the diffusion probability model as claimed in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: The method comprises one or more programs executed by one or more processors of an electronic device, wherein the one or more programs comprise instructions for executing the image generation method based on the diffusion probability model as claimed in any one of claims 1 to 8.

Images