A diffusion model training method, device, equipment, medium and video generation method
By extracting frames and Gaussian noise processing on the video, the diffusion model is constructed, which solves the problem of insufficient time consistency in the generation of noisy image videos, and improves the viewing and experience of the video.
Patent Information
- Application Number
- CN202411272970.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-12
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-09-12
Smart Images

Figure CN118781453B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of video generation, and in particular to a diffusion model training method, device, equipment, medium and video generation method. Background Art
[0002] Video generation refers to the use of algorithms and models to create new, playable video content. It generates new videos related to inputs such as images, text, or existing video clips. The purpose of video generation is to enable computers to understand and create dynamic visual content.
[0003] With the rapid development of stable diffusion (a model that generates high-quality images by gradually removing noise), these models have been widely used in the field of generating high-quality images from text. However, when these models are applied to the video domain, ensuring the temporal consistency between video frames remains a huge challenge. Therefore, a solution is urgently needed for video generation based on noisy images. Summary of the invention
[0004] The embodiments of the present invention solve the technical problem of low temporal consistency when generating videos based on noise images in the prior art by providing a diffusion model training method, device, equipment, medium and video generation method, and achieve the technical effect of improving temporal consistency when generating videos based on noise images.
[0005] In a first aspect, the present invention provides a diffusion model training method, the method comprising:
[0006] Extract frames from the video to be processed, obtain several consecutive images to be processed, and divide each image to be processed into several regions of interest;
[0007] According to a preset adding rule, Gaussian noise is added to several regions of interest in each image to be processed respectively and continuously to obtain a noise image set corresponding to each image to be processed, wherein the noise image set includes several noise images;
[0008] Perform diffusion training based on a number of noise image sets, wherein the diffusion training includes noise prediction training and video frame prediction training;
[0009] Among them, noise prediction training includes:
[0010] Constructing a noise neural network model to be trained, wherein the input of the noise neural network model to be trained is an image containing noise, and the output is the predicted noise;
[0011] Inputting each noise image in a number of noise image sets into the noise neural network model to be trained, and updating the noise neural network model to be trained according to the predicted noise and the added noise of each region of interest in each noise image to obtain a target noise neural network model;
[0012] Among them, video frame prediction training includes:
[0013] Constructing a video frame prediction model to be trained, wherein the input of the video frame prediction model to be trained is two consecutive frames of images containing noise, and the output is the predicted noise difference;
[0014] Inputting a set of noise images from a set of two consecutive noise images into the video frame prediction model to be trained, and updating the video frame prediction model to be trained according to actual noise differences and predicted noise differences of the several sets of noise images to obtain a target video frame prediction model, wherein the set of noise images includes two noise images;
[0015] According to the target noise neural network model and the target video frame prediction model, a video generation diffusion model is obtained.
[0016] Furthermore, a set of noise images in a set of noise images of two consecutive frames is inputted into the video frame prediction model to be trained, and the video frame prediction model to be trained is updated according to the actual noise difference and the predicted noise difference of the several sets of noise images to obtain the target video frame prediction model, including:
[0017] Select any two adjacent noise image sets;
[0018] Randomly select one noise image from each of the noise image sets of two adjacent frames to form a group of noise images;
[0019] Determine the Gaussian noise added to each noise image in the group of noise images according to the Gaussian noise added to a plurality of regions of interest of the noise image;
[0020] determining an actual noise difference of the group of noise images according to a difference between Gaussian noises added to two noise images in the group of noise images;
[0021] After inputting the group of noise images into the video frame prediction model to be trained, determining the predicted noise difference corresponding to the group of noise images;
[0022] Adjusting a neural network parameter of a prediction model for a video frame to be trained according to a predicted noise difference corresponding to the group of noise images and an actual noise difference of the group of noise images;
[0023] When the preset number of iterations is reached or the first preset output standard is reached, the latest neural network parameters of the video frame prediction model to be trained are saved, and the target video frame prediction model is obtained.
[0024] Furthermore, Gaussian noise is added to several regions of interest in each image to be processed respectively and continuously according to a preset adding rule, so as to obtain a noise image set corresponding to each image to be processed, including:
[0025] For each region of interest in each image to be processed, the variance of the Gaussian noise added to the region of interest each time is determined according to the number of pixels, the region contour, the region flatness and the number of additions of each region of interest;
[0026] Determine a noise image corresponding to the image to be processed according to the variance of the Gaussian noise of each region of interest in the image to be processed when added at the same time;
[0027] A noise image set corresponding to the image to be processed is formed according to a plurality of noise images corresponding to the addition of Gaussian noise to the image to be processed for a plurality of times.
[0028] Furthermore, the formula for adding Gaussian noise to each region of interest includes:
[0029]
[0030] or,
[0031]
[0032] in, The variance of the Gaussian noise added to the region of interest, , are weighted empirical coefficients, is the sensitive empirical coefficient, is the number of pixels in the i-th region of interest, is the area contour of the i-th region of interest, is the regional flatness of the i-th region of interest, n is the number of additions, To preset the standard number of pixels, It is the preset standard area contour.
[0033] Furthermore, each noise image in the noise image set is input into the noise neural network model to be trained, and the noise neural network model to be trained is updated according to the predicted noise and the added noise of each region of interest in each noise image to obtain a target noise neural network model, including:
[0034] Inputting any noise image of any noise image set into the noise neural network model to be trained, and obtaining the predicted noise of each region of interest of the noise image;
[0035] According to the predicted noise of each region of interest in the noisy image and the added noise of each region of interest, the neural network parameters of the noise neural network model to be trained are updated;
[0036] After the neural network parameters of the noise neural network model to be trained are updated according to the predicted noise of all the regions of interest in the noise image and the added noise, the Gaussian noise added to the noise image is determined according to the predicted noise of each region of interest in the noise image;
[0037] The neural network parameters of the noise neural network model to be trained are updated again according to the Gaussian noise added to the noise image and the predicted noise corresponding to the noise image;
[0038] When the preset number of iterations is reached or the second preset output standard is reached, the latest neural network parameters of the noise neural network model to be trained are saved, and the target noise neural network model is obtained.
[0039] Furthermore, each image to be processed is divided into several regions of interest, including:
[0040] According to the size of each image to be processed, each image to be processed is divided into a number of regions of interest, wherein the sizes of the regions of interest are the same.
[0041] In a second aspect, the present invention provides a video generation method, comprising:
[0042] Get the target noise image;
[0043] Input the target noise image into the video generation diffusion model to obtain several consecutive denoised images;
[0044] Synthesize the target video based on several consecutive denoised images.
[0045] In a third aspect, the present invention provides a diffusion model training device, the device comprising:
[0046] The video processing module is used to extract frames from the video to be processed, obtain a number of continuous images to be processed, and divide each image to be processed into a number of regions of interest;
[0047] A noise adding module, used for adding Gaussian noise to several regions of interest in each image to be processed respectively and continuously according to a preset adding rule, so as to obtain a noise image set corresponding to each image to be processed, wherein the noise image set includes several noise images;
[0048] A model training module, used for performing diffusion training based on a plurality of noise image sets, wherein the diffusion training includes noise prediction training and video frame prediction training;
[0049] Among them, noise prediction training includes:
[0050] Constructing a noise neural network model to be trained, wherein the input of the noise neural network model to be trained is an image containing noise, and the output is the predicted noise;
[0051] Inputting each noise image in a number of noise image sets into the noise neural network model to be trained, and updating the noise neural network model to be trained according to the predicted noise and the added noise of each region of interest in each noise image to obtain a target noise neural network model;
[0052] Among them, video frame prediction training includes:
[0053] Constructing a video frame prediction model to be trained, wherein the input of the video frame prediction model to be trained is two consecutive frames of images containing noise, and the output is the predicted noise difference;
[0054] Inputting a set of noise images from a set of two consecutive noise images into the video frame prediction model to be trained, and updating the video frame prediction model to be trained according to actual noise differences and predicted noise differences of the several sets of noise images to obtain a target video frame prediction model, wherein the set of noise images includes two noise images;
[0055] The synthesis module is used to obtain a video generation diffusion model according to a target noise neural network model and a target video frame prediction model.
[0056] In a fourth aspect, the present invention provides an electronic device, comprising:
[0057] processor;
[0058] a memory for storing processor-executable instructions;
[0059] The processor is configured to execute to implement a diffusion model training method as provided in the first aspect.
[0060] In a fifth aspect, the present invention provides a non-temporary computer-readable storage medium, which, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to implement a diffusion model training method.
[0061] One or more technical solutions provided in the embodiments of the present invention have at least the following technical effects or advantages:
[0062] The present invention trains the training video frame prediction model. When the target video frame prediction model is obtained, it means that the Gaussian noise difference of two consecutive input frames is within a certain range, indicating that the two consecutive frames have temporal consistency. By making the two consecutive frames have temporal consistency, the video synthesized based on the noise picture can have a stronger visual experience, thereby improving the user experience.
[0063] The present invention also provides a method for determining the amount of Gaussian noise added to two regions of interest. When the accuracy of the model needs to be ensured, the amount of Gaussian noise added to the same region of interest is determined by the number of times the noise is added each time. The variance of the Gaussian noise added to the region of interest can ensure that the variance of the Gaussian noise of the region of interest at the same position of two consecutive adjacent noise images grows at a multiple rate. In other words, for the model to be trained, the more uniform the change, the more conducive to model training. When it is necessary to ensure the training speed of the model, by adding according to the exponential, the processing of the Gaussian noise added to the region of interest can be completed as soon as possible, so that the image to be processed can be iterated to a pure noise image as soon as possible.
[0064] When adding Gaussian noise to a region of interest, the present invention introduces parameters such as the number of pixel points, the region contour of the region of interest, and the region flatness of the region of interest, so that each time Gaussian noise is added to the region of interest, the variance of the Gaussian noise added to the region of interest can be accurately determined. Compared with the uniform addition in the prior art, the present invention divides the region and determines the variance of the Gaussian noise added to each region, so that Gaussian noise can be added according to the differences between the regions of interest in each image. When the model is subsequently trained using the regions of interest, more input details can be provided for the model under the condition of the same input image, and the region of interest and the noise image are used for secondary updating, which facilitates model training and improves the accuracy of model training.
[0065] The present invention also provides a video generation method, which can judge whether the predicted noise map has temporal consistency based on a noise map through a target video frame prediction model, and the video synthesized in this way has temporal consistency. BRIEF DESCRIPTION OF THE DRAWINGS
[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0067] Figure 1 A flow chart of a diffusion model training method provided by the present invention;
[0068] Figure 2 A schematic diagram of a flow chart of a video generation method provided by the present invention;
[0069] Figure 3 A schematic diagram of dividing the region of interest of the image to be processed provided by the present invention;
[0070] Figure 4 A schematic diagram of adding Gaussian noise to an image to be processed provided by the present invention;
[0071] Figure 5 Schematic diagram of the noise image set provided by the present invention
[0072] Figure 6 Schematic diagram of the process of obtaining several denoised images provided by the present invention
[0073] Figure 7 A structural schematic diagram of a diffusion model training device provided by the present invention;
[0074] Figure 8 The present invention provides a schematic structural diagram of an electronic device. DETAILED DESCRIPTION
[0075] The embodiment of the present invention solves the technical problem of low temporal consistency when generating videos based on noise images in the prior art by providing a diffusion model training method.
[0076] The technical solution of the embodiment of the present invention is to solve the above technical problems, and the overall idea is as follows:
[0077] A diffusion model training method, the method comprising: extracting frames of a video to be processed, obtaining a plurality of continuous images to be processed, and dividing each image to be processed into a plurality of regions of interest; adding Gaussian noise to the plurality of regions of interest in each image to be processed respectively and continuously according to a preset adding rule, obtaining a noise image set corresponding to each image to be processed, wherein the noise image set comprises a plurality of noise images; performing diffusion training based on the plurality of noise image sets, wherein the diffusion training comprises noise prediction training and video frame prediction training; wherein the noise prediction training comprises: constructing a noise neural network model to be trained, wherein the input of the noise neural network model to be trained is an image containing noise, and the output is predicted noise; inputting each noise image in the plurality of noise image sets into the noise neural network model to be trained, and outputting the predicted noise according to the noise image set; According to the predicted noise and added noise of each area of interest in each noise image, the noise neural network model to be trained is updated to obtain a target noise neural network model; wherein, the video frame prediction training includes: constructing a video frame prediction model to be trained, wherein the input of the video frame prediction model to be trained is two consecutive frames of images containing noise, and the output is a predicted noise difference; inputting a group of noise images in a set of two consecutive frames of noise images into the video frame prediction model to be trained, and updating the video frame prediction model to be trained according to the actual noise difference and the predicted noise difference of several groups of noise images to obtain a target video frame prediction model, wherein a group of noise images includes two noise images; according to the target noise neural network model and the target video frame prediction model, a video generation diffusion model is obtained.
[0078] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0079] First of all, the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.
[0080] The present invention provides Figure 1 A diffusion model training method is shown, comprising steps S11-S14, wherein the noise picture and noise image mentioned in the present invention may refer to a picture or image containing Gaussian noise:
[0081] Regarding step S11, frames are extracted from the video to be processed to obtain a number of continuous images to be processed, and each image to be processed is divided into a number of regions of interest.
[0082] The video to be processed is a video with frame extraction requirements. The duration of the video to be processed can be 1s, 2s, ..60s, etc., which can be determined according to actual needs. The longer the duration of the video to be processed, the better the training effect of the model, but the longer the training time required.
[0083] Generally speaking, a video has 30 frames, 60 frames, or 120 frames, that is, a one-second video is composed of 30, 60, or 120 consecutive pictures. The number of pictures that can be extracted from the video can be determined based on the specific situation of the video to be processed.
[0084] It should be noted that in order to ensure the training effect of the model, the video to be processed can be a video with a unified scene (that is, a video with an unchanged scene. The purpose of the present invention is to provide a model for synthesizing videos based on noise images and to ensure the temporal consistency of the synthesized video. Therefore, the selection of videos with scenes that do not change drastically is more conducive to model training).
[0085] After the video to be processed is subjected to frame extraction processing, each image to be processed can be divided into a number of regions of interest according to the size of each image to be processed, wherein the sizes of the regions of interest are the same.
[0086] like Figure 3 As shown, Figure 3 For an image extracted from a certain video to be processed, the image to be processed is divided into a number of regions of interest, and the size of each region of interest is the same.
[0087] Regarding step S12, Gaussian noise is added to several regions of interest in each image to be processed respectively and continuously according to a preset adding rule to obtain a noise image set corresponding to each image to be processed, and the noise image set includes several noise images.
[0088] After each image to be processed is divided, Gaussian noise can be added according to each region of interest of each image to be processed, including:
[0089] For each region of interest in each image to be processed, the variance of the Gaussian noise when the region of interest is added each time is determined according to the number of pixels, region contour, region flatness and the number of additions of each region of interest.
[0090] The formulas for adding Gaussian noise to a pixel in an image include:
[0091]
[0092] in, is the pixel with coordinates (i, j) in the image to be processed, The pixel with coordinates (i, j) in the processed image is added with Gaussian noise. is a Gaussian (normal) distributed random variable with added Gaussian noise, where the Gaussian noise appears as random fluctuations with mean 0 and variance σ.
[0093] Determine a noise image corresponding to the image to be processed according to the variance of the Gaussian noise of each region of interest in the image to be processed when added at the same time;
[0094] After dividing the regions, the present invention adds Gaussian noise to each image in turn, as shown in Figure 4 ( Figure 4 The left side is the image before noise addition, and the right side is the image after noise addition). After obtaining the image to be processed, the present invention adds Gaussian noise once to each region of interest, and thus obtains a noise image corresponding to the image to be processed.
[0095] Gaussian noise is added to each region of interest, including:
[0096]
[0097] When the accuracy of the model needs to be ensured, the same area of interest is added each time to determine the number of The variance of the Gaussian noise added to the region of interest can ensure that the variance of the Gaussian noise in the region of interest at the same position of two consecutive adjacent noise images increases at a multiple rate. In other words, for the model to be trained, the more uniform the change, the more conducive to model training.
[0098]
[0099] in, The variance of the Gaussian noise added to the region of interest, , are weighted empirical coefficients, is the sensitive empirical coefficient, is the number of pixels in the i-th region of interest, is the area contour of the i-th region of interest, is the regional flatness of the i-th region of interest, n is the number of additions, To preset the standard number of pixels, It is the preset standard area contour.
[0100] When the training speed of the model needs to be guaranteed, the Gaussian noise added to the region of interest can be processed as quickly as possible by adding it exponentially, so that the image to be processed can be iterated into a pure noise image as quickly as possible.
[0101] In addition, when adding Gaussian noise to the region of interest, the present invention introduces parameters such as the number of pixels, the regional contour of the region of interest, and the regional flatness of the region of interest, so that each time Gaussian noise is added to the region of interest, the variance of the Gaussian noise added to the region of interest can be accurately determined. Compared with the uniform addition in the prior art, the present invention divides the region and determines the variance of the Gaussian noise added to each region, so that Gaussian noise can be accurately added according to the differences in each region of interest in each image. When the model is subsequently trained using each region of interest, more input details can be provided for the model under the condition of the same input image, which facilitates model training and improves the accuracy of model training.
[0102] The region contour can be determined based on parameters such as the perimeter, area or shape factor of each region of interest by extracting the contour of each region of interest based on an edge detection algorithm.
[0103] The regional flatness can be analyzed by the texture of the region through the gray level co-occurrence matrix (GLCM) and the flatness of each interested region can be determined by indicators such as contrast, entropy and uniformity. The preset standard pixel number can be the number of pixels of the region of interest with the maximum number of pixels in each image to be processed or the mean value of the pixels of each image to be processed. The preset standard regional contour can be the regional contour of the region of interest with the maximum regional contour in each image to be processed, or the mean value of the regional contour of each image to be processed. The specific method can be determined according to the actual situation.
[0104] A noise image set corresponding to the image to be processed is formed according to a plurality of noise images corresponding to the addition of Gaussian noise to the image to be processed for a plurality of times.
[0105] After adding noise to each region of interest of the image to be processed several times, a noise image set corresponding to the image to be processed can be obtained. Figure 5 For reference, Figure 5 In the above, (a), (b), (c), ..., (n) are images obtained by adding noise once, twice, ... n times, respectively. (a), (b), (c), ..., (n) together constitute a noise image set (the noise image set may include the image to be processed, which can be regarded as a noise image with zero noise).
[0106] Regarding step S13, diffusion training is performed based on a number of noise image sets, wherein the diffusion training includes noise prediction training and video frame prediction training.
[0107] After processing each to-be-processed picture according to steps S11-S12 and obtaining a noise image set corresponding to each to-be-processed picture, diffusion training may be performed, specifically including noise prediction training and video frame prediction training.
[0108]
About noise prediction training
[0109] A noise neural network model to be trained is constructed, wherein the input of the noise neural network model to be trained is an image containing noise, and the output is the predicted noise; each noise image in a number of noise image sets is input into the noise neural network model to be trained, and according to the predicted noise and added noise of each region of interest in each noise image, the noise neural network model to be trained is updated to obtain a target noise neural network model.
[0110] Specifically include:
[0111] The noise neural network model to be trained provided by the present invention mainly includes an input layer, a convolution layer, a downsampling layer, a fully connected layer and an output layer. Adaptive initial parameters can be set in each layer (for example, the number of convolution layers and downsampling layers, the size of the convolution kernel, the number of convolution kernels, and the reduction amplitude of the pooling layer, and can also include parameters such as learning rate and batch size), and initialized.
[0112] After the noise neural network model to be trained is constructed, any noise image of any noise image set can be input into the noise neural network model to be trained, and the predicted noise of each region of interest of the noise image can be obtained.
[0113] It can be understood that since the present invention divides each image to be processed into regions of interest, the noise image corresponding to the image to be processed is also divided into regions of interest. Adaptive initial parameters can be set in the noise neural network model to be trained so that the noise neural network model to be trained can respectively identify the predicted noise of each region of interest in each noise image.
[0114] After obtaining the predicted noise of each region of interest, the neural network parameters of the noisy neural network model to be trained are updated (for example, weights, biases, learning rates, batch sizes, optimizers, momentum, etc.) according to the predicted noise of each region of interest in the noisy image and the added noise of each region of interest (which can be determined according to the formula provided above).
[0115] By continuously iteratively updating the neural network parameters, the error between the predicted noise and the corresponding added noise is reduced.
[0116] After the neural network parameters of the noise neural network model to be trained are updated according to the predicted noise of all the regions of interest in the noise image and the added noise, the Gaussian noise added to the noise image is determined according to the predicted noise of each region of interest in the noise image.
[0117] The neural network parameters of the noise neural network model to be trained are updated again according to the Gaussian noise added to the noise image and the predicted noise corresponding to the noise image.
[0118] It can be understood that the present invention updates the neural network parameters based on the predicted noise and added noise of each region of interest, thereby improving data utilization and model training accuracy. It also determines the added Gaussian noise of each noise image based on the added noise of each region of interest, and compares the added Gaussian noise of each noise image with the predicted noise for further updating. This means that without adding additional images, data utilization is improved, a second update is performed, and the accuracy of model training is improved.
[0119] When the preset number of iterations is reached or the second preset output standard is reached, the latest neural network parameters of the noise neural network model to be trained are saved, and the target noise neural network model is obtained.
[0120] The second preset output standard may be that the mean of the difference between several groups of predicted noise and added noise is less than the first threshold, and the preset number of iterations may be determined according to actual conditions.
[0121]
Video frame prediction training
[0122] A video frame prediction model to be trained is constructed, wherein the input of the video frame prediction model to be trained is two consecutive frames of images containing noise, and the output is the predicted noise difference; a group of noise images in a set of two consecutive frames of noise images is input into the video frame prediction model to be trained, and the video frame prediction model to be trained is updated according to the actual noise difference and the predicted noise difference of several groups of noise images to obtain a target video frame prediction model, wherein a group of noise images includes two noise images.
[0123] Specifically include:
[0124] Select any noise image set of two adjacent frames, and select any noise image from each of the noise image sets of two adjacent frames to form a group of noise images.
[0125] For example, the image to be processed E and the image to be processed F are two adjacent frames of images, E corresponds to the noise image set e, F corresponds to the noise image set f, and each of the noise image set e and the noise image set f includes 100 noise images. A noise image can be selected from the noise image set e, and a noise image can be selected from the noise image set f, and a group of noise images can be formed based on these two images.
[0126] The Gaussian noise added to each noise image in the group of noise images may be determined according to the Gaussian noise added to a plurality of regions of interest of the noise image.
[0127] After the Gaussian noise added to each noise image is determined, the actual noise difference of the group of noise images can be determined according to the difference between the Gaussian noise added to two noise images in the group of noise images.
[0128] After determining the actual noise difference of the group of noise images, the group of noise images may be input into the video frame prediction model to be trained to determine the predicted noise difference corresponding to the group of noise images.
[0129] After the predicted noise difference corresponding to the group of noise images is determined, the neural network parameters of the video frame prediction model to be trained can be adjusted according to the predicted noise difference corresponding to the group of noise images and the actual noise difference of the group of noise images.
[0130] When the preset number of iterations is reached or the first preset output standard is reached, the latest neural network parameters of the video frame prediction model to be trained are saved, and the target video frame prediction model is obtained.
[0131] The first preset output standard may be that the average of the differences between a number of predicted noise differences and actual noise differences is less than a second threshold value, and the preset number of iterations may also be determined based on actual conditions.
[0132] It should be understood that the purpose of the training model of the present invention is to obtain a video with temporal consistency based on a noisy image (that is, the previous and next frames of the image are related, and there is no direct jump from one scene to another, and the movement of objects in the video is continuous).
[0133] The present invention trains the to-be-trained video frame prediction model. When the target video frame prediction model is obtained, it means that the Gaussian noise difference between two consecutive input frames is within a certain range, indicating that the two consecutive frames have temporal consistency.
[0134] Regarding step S14, a video generation diffusion model is obtained according to the target noise neural network model and the target video frame prediction model.
[0135] The target noise neural network model and the target video frame prediction model are obtained, and the two models can be combined into a video generation diffusion model.
[0136] In addition, video prediction models such as long short-term memory networks, gated recurrent units, 3D convolutional neural networks, etc. can be added to the video generation diffusion model to predict the next frame of the noise image based on the noise image.
[0137] Please refer to the following description for the method of generating videos based on the video generation diffusion model.
[0138] Based on the above-mentioned video generation diffusion model, the present invention also provides Figure 2The video generation method shown includes steps S21-S23, please refer to Figure 6 :
[0139] Regarding step S21, a target noise image is obtained.
[0140] You can get any noise image.
[0141] Regarding step S22, a target noise image is input into the video generation diffusion model to obtain a number of consecutive denoised images.
[0142] Please combine Figure 6 For reference, the video prediction model in the video diffusion model can be generated based on the video (the video prediction model is Figure 6 The video frame model in is used to predict the next noise picture (h) corresponding to the noise picture (g). The noise picture (g) and the noise picture (h) are input into the target video frame prediction model to predict the noise difference between the noise picture (g) and the noise picture (h). If the difference is within the threshold range, it means that the noise picture (g) and the noise picture (h) have temporal consistency. The noise picture (g) and the noise picture (h) are respectively input into the target noise neural network model (i.e. Figure 6 In the noise model in ), the predicted noise of the noise image (g) and the noise image (h) is obtained. According to the predicted noise, denoising is performed iteratively to obtain the noise-free images G and H corresponding to the noise image (g) and the noise image (h).
[0143] And when the noise difference between the predicted noise picture (g) and the noise picture (h) is within the threshold range, the predicted noise picture (i) of the next frame of the noise picture (h) can be predicted based on the noise picture (h), and the above process is repeated continuously.
[0144] Regarding step S23, a target video is synthesized based on a plurality of consecutive denoised images.
[0145] Repeat the above steps continuously, determine the number of noise-free images to be obtained according to actual needs, and synthesize several consecutive denoised images into a target video.
[0146] In summary, the present invention extracts frames from a video to be processed to obtain a number of continuous images to be processed, and divides each image to be processed into a number of regions of interest; according to a preset adding rule, Gaussian noise is respectively and continuously added to a number of regions of interest in each image to be processed to obtain a noise image set corresponding to each image to be processed, wherein the noise image set includes a number of noise images; diffusion training is performed based on the number of noise image sets, wherein the diffusion training includes noise prediction training and video frame prediction training; wherein the noise prediction training includes: constructing a noise neural network model to be trained, wherein the input of the noise neural network model to be trained is an image containing noise, and the output is predicted noise; each noise image in the number of noise image sets is input into the noise neural network model to be trained, and according to each noise image, a noise prediction training method is used to predict the noise of the image. The predicted noise and added noise of each area of interest in the noise image are used to update the noise neural network model to be trained to obtain a target noise neural network model; wherein, the video frame prediction training includes: constructing a video frame prediction model to be trained, wherein the input of the video frame prediction model to be trained is two consecutive frames of noise-containing images, and the output is a predicted noise difference; inputting a group of noise images in a set of noise images of two consecutive frames into the video frame prediction model to be trained, and updating the video frame prediction model to be trained according to the actual noise difference and the predicted noise difference of several groups of noise images to obtain a target video frame prediction model, wherein a group of noise images includes two noise images; according to the target noise neural network model and the target video frame prediction model, a video generation diffusion model is obtained. The present invention trains the video frame prediction model to be trained, and when the target video frame prediction model is obtained, it means that the Gaussian noise difference of the input two consecutive frames of pictures is within a certain range, indicating that the two consecutive frames of pictures have temporal consistency. By making the two consecutive frames of pictures have temporal consistency, the video synthesized based on the noise picture can have a strong perception, thereby improving the user experience.
[0147] The present invention also provides a method for determining the amount of Gaussian noise added to two regions of interest. When the accuracy of the model needs to be ensured, the amount of Gaussian noise added to the same region of interest is determined by the number of times the noise is added each time. The variance of the Gaussian noise added to the region of interest can ensure that the variance of the Gaussian noise of the region of interest at the same position of two consecutive adjacent noise images grows at a multiple rate. In other words, for the model to be trained, the more uniform the change, the more conducive to model training. When it is necessary to ensure the training speed of the model, by adding according to the exponential, the processing of the Gaussian noise added to the region of interest can be completed as soon as possible, so that the image to be processed can be iterated to a pure noise image as soon as possible.
[0148] When adding Gaussian noise to a region of interest, the present invention introduces parameters such as the number of pixel points, the region contour of the region of interest, and the region flatness of the region of interest, so that each time Gaussian noise is added to the region of interest, the variance of the Gaussian noise added to the region of interest can be accurately determined. Compared with the uniform addition in the prior art, the present invention divides the region and determines the variance of the Gaussian noise added to each region, so that Gaussian noise can be added according to the differences between the regions of interest in each image. When the model is subsequently trained using the regions of interest, more input details can be provided for the model under the condition of the same input image, and the region of interest and the noise image are used for secondary updating, which facilitates model training and improves the accuracy of model training.
[0149] The present invention also provides a video generation method, which can judge whether the predicted noise map has temporal consistency based on a noise map through a target video frame prediction model, and the video synthesized in this way has temporal consistency.
[0150] Based on the same inventive concept, the present invention also provides Figure 7 A diffusion model training device is shown, the device comprising:
[0151] The video processing module 71 is used to extract frames of the video to be processed, obtain a number of continuous images to be processed, and divide each image to be processed into a number of regions of interest;
[0152] A noise adding module 72 is used to add Gaussian noise to several regions of interest in each image to be processed respectively and continuously according to a preset adding rule, so as to obtain a noise image set corresponding to each image to be processed, wherein the noise image set includes several noise images;
[0153] A model training module 73, configured to perform diffusion training based on a plurality of noise image sets, wherein the diffusion training includes noise prediction training and video frame prediction training;
[0154] Among them, noise prediction training includes:
[0155] Constructing a noise neural network model to be trained, wherein the input of the noise neural network model to be trained is an image containing noise, and the output is the predicted noise;
[0156] Inputting each noise image in a number of noise image sets into the noise neural network model to be trained, and updating the noise neural network model to be trained according to the predicted noise and the added noise of each region of interest in each noise image to obtain a target noise neural network model;
[0157] Among them, video frame prediction training includes:
[0158] Constructing a video frame prediction model to be trained, wherein the input of the video frame prediction model to be trained is two consecutive frames of images containing noise, and the output is the predicted noise difference;
[0159] Inputting a set of noise images from a set of two consecutive noise images into the video frame prediction model to be trained, and updating the video frame prediction model to be trained according to actual noise differences and predicted noise differences of the several sets of noise images to obtain a target video frame prediction model, wherein the set of noise images includes two noise images;
[0160] The synthesis module 74 is used to obtain a video generation diffusion model according to the target noise neural network model and the target video frame prediction model.
[0161] Based on the same inventive concept, the present invention also provides Figure 8 An electronic device as shown includes:
[0162] Processor 81;
[0163] a memory 82 for storing instructions executable by the processor 81;
[0164] The processor 81 is configured to execute to implement a diffusion model training method as provided above.
[0165] Based on the same inventive concept, the present invention also provides a non-temporary computer-readable storage medium. When the instructions in the storage medium are executed by the processor 81 of the electronic device, the electronic device can implement a diffusion model training method as provided above.
[0166] Since the electronic device introduced in this embodiment is an electronic device used to implement the information processing method in the embodiment of the present invention, based on the information processing method introduced in the embodiment of the present invention, a person skilled in the art can understand the specific implementation of the electronic device of this embodiment and its various variations, so how the electronic device implements the method in the embodiment of the present invention is not described in detail here. As long as the electronic device used by a person skilled in the art to implement the information processing method in the embodiment of the present invention, it belongs to the scope of protection of the present invention.
[0167] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0168] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0169] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0170] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0171] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.
[0172] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A diffusion model training method, It is characterized in that The method comprises: Extract frames from the video to be processed, obtain several consecutive images to be processed, and divide each image to be processed into several regions of interest; According to a preset adding rule, Gaussian noise is added to several regions of interest in each image to be processed respectively and continuously, and a noise image set corresponding to each image to be processed is obtained, wherein the noise image set includes several noise images, including: for each region of interest in each image to be processed, according to the number of pixels, region contour, region flatness and number of additions of each region of interest, the variance of Gaussian noise added to the region of interest each time is determined; according to the variance of Gaussian noise of each region of interest in the image to be processed when the same addition is made, a noise image corresponding to the image to be processed is determined; according to several noise images corresponding to the addition of Gaussian noise to the image to be processed several times, a noise image set corresponding to the image to be processed is formed, wherein the formula for adding Gaussian noise to each region of interest includes: or, in, The variance of the Gaussian noise added to the region of interest, , are weighted empirical coefficients, is the sensitive empirical coefficient, is the number of pixels in the i-th region of interest, is the area contour of the i-th region of interest, is the regional flatness of the i-th region of interest, n is the number of additions, To preset the standard number of pixels, It is a preset standard regional contour, wherein the contour of each region of interest is extracted based on the edge detection algorithm, and the regional contour of the region of interest is determined based on the perimeter, area or shape factor of each region of interest; the texture of the region of interest is analyzed based on the gray level co-occurrence matrix, and the regional flatness of the region of interest is determined by contrast, entropy and uniformity, wherein the number of additions refers to the number of times Gaussian noise is added; Performing diffusion training based on a plurality of noise image sets, wherein the diffusion training includes noise prediction training and video frame prediction training; Wherein, the noise prediction training includes: Constructing a noise neural network model to be trained, wherein the input of the noise neural network model to be trained is an image containing noise, and the output is predicted noise; Inputting each noise image in a plurality of noise image sets into the noise neural network model to be trained, and updating the noise neural network model to be trained according to the predicted noise and the added noise of each region of interest in each noise image to obtain a target noise neural network model; The video frame prediction training includes: Constructing a video frame prediction model to be trained, wherein the input of the video frame prediction model to be trained is two consecutive frames of images containing noise, and the output is a predicted noise difference; Inputting a set of noise images from a set of two consecutive noise images into the video frame prediction model to be trained, and updating the video frame prediction model to be trained according to actual noise differences and predicted noise differences of the multiple sets of noise images to obtain a target video frame prediction model, wherein the set of noise images includes two noise images; A video generation diffusion model is obtained according to the target noise neural network model and the target video frame prediction model.
2. A diffusion model training method as claimed in claim 1, It is characterized in that The step of inputting a set of noise images from a set of noise images of two consecutive frames into the video frame prediction model to be trained, and updating the video frame prediction model to be trained according to actual noise differences and predicted noise differences of the plurality of sets of noise images to obtain a target video frame prediction model comprises: Select any two adjacent noise image sets; Randomly select one noise image from each of the noise image sets of two adjacent frames to form a group of noise images; Determine the Gaussian noise added to each noise image in the group of noise images according to the Gaussian noise added to a plurality of regions of interest of the noise image; determining an actual noise difference of the group of noise images according to a difference between Gaussian noises added to two noise images in the group of noise images; After inputting the group of noise images into the video frame prediction model to be trained, determining the predicted noise differences corresponding to the group of noise images; Adjusting the neural network parameters of the to-be-trained video frame prediction model according to the predicted noise difference corresponding to the group of noise images and the actual noise difference of the group of noise images; When the preset number of iterations is reached or the first preset output standard is reached, the latest neural network parameters of the video frame prediction model to be trained are saved, and the target video frame prediction model is obtained.
3. A diffusion model training method as claimed in claim 1, It is characterized in that The step of inputting each noise image in a plurality of noise image sets into the noise neural network model to be trained, and updating the noise neural network model to be trained according to the predicted noise and the added noise of each region of interest in each noise image to obtain a target noise neural network model comprises: Inputting any noise image of any noise image set into the noise neural network model to be trained, and obtaining the predicted noise of each region of interest of the noise image; According to the predicted noise of each region of interest in the noisy image and the added noise of each region of interest, updating the neural network parameters of the noise neural network model to be trained; After the neural network parameters of the noise neural network model to be trained are updated according to the predicted noise of all the regions of interest in the noise image and the added noise, the Gaussian noise added to the noise image is determined according to the predicted noise of each region of interest in the noise image; The neural network parameters of the noise neural network model to be trained are updated again according to the Gaussian noise added to the noise image and the predicted noise corresponding to the noise image; When the preset number of iterations is reached or the second preset output standard is reached, the latest neural network parameters of the noise neural network model to be trained are saved, and the target noise neural network model is obtained.
4. A diffusion model training method as claimed in claim 1, It is characterized in that Each image to be processed is divided into several regions of interest, including: According to the size of each image to be processed, each image to be processed is divided into a number of regions of interest, wherein the sizes of the regions of interest are the same.
5. A diffusion model training device, It is characterized in that A diffusion model training method applied to any one of claims 1 to 4, the device comprising: The video processing module is used to extract frames from the video to be processed, obtain a number of continuous images to be processed, and divide each image to be processed into a number of regions of interest; A noise adding module, used for adding Gaussian noise to several regions of interest in each image to be processed respectively and continuously according to a preset adding rule, so as to obtain a noise image set corresponding to each image to be processed, wherein the noise image set includes several noise images; A model training module, used for performing diffusion training based on a plurality of noise image sets, wherein the diffusion training includes noise prediction training and video frame prediction training; Wherein, the noise prediction training includes: Constructing a noise neural network model to be trained, wherein the input of the noise neural network model to be trained is an image containing noise, and the output is predicted noise; Inputting each noise image in a plurality of noise image sets into the noise neural network model to be trained, and updating the noise neural network model to be trained according to the predicted noise and the added noise of each region of interest in each noise image to obtain a target noise neural network model; The video frame prediction training includes: Constructing a video frame prediction model to be trained, wherein the input of the video frame prediction model to be trained is two consecutive frames of images containing noise, and the output is a predicted noise difference; Inputting a set of noise images from a set of two consecutive noise images into the video frame prediction model to be trained, and updating the video frame prediction model to be trained according to actual noise differences and predicted noise differences of the multiple sets of noise images to obtain a target video frame prediction model, wherein the set of noise images includes two noise images; A synthesis module is used to obtain a video generation diffusion model according to the target noise neural network model and the target video frame prediction model.
6. An electronic device, It is characterized in that include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute to implement a diffusion model training method as described in any one of claims 1 to 4.
7. A non-transitory computer-readable storage medium, It is characterized in that When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to implement a diffusion model training method as described in any one of claims 1 to 4.
Citation Information
Patent Citations
Single-frame image super-resolution processing method based on diffusion model
CN113177882A
Image Enhancement via Iterative Refinement based on Machine Learning Models
US20230067841A1
