Video generation method, device and equipment based on diffusion model

Abstract

The present invention discloses a method, device and equipment for video generation based on a diffusion model, including: obtaining a plurality of frame-drawing pictures; constructing a plurality of picture groups; mapping the picture groups in the initial data space to the latent space; constructing a diffusion model to be trained; adding noise to each latent variable in the picture group; inputting the picture group into the diffusion model to be trained, obtaining a plurality of predicted noises and a plurality of predicted picture differences corresponding to each picture group, and a plurality of predicted picture group differences corresponding to the picture groups; adjusting the neural network parameters of the diffusion model to be trained according to the plurality of predicted noises, the plurality of predicted picture differences and the plurality of predicted picture group differences, and using the diffusion model to be trained as the target diffusion model when the preset training requirements are met. The present invention belongs to the field of video prediction. The present invention can make the video predicted based on the diffusion model maintain time consistency.

Description

Technical Field

[0001] The present invention relates to the field of video prediction, and in particular to a method, device and equipment for generating video based on a diffusion model. Background Art

[0002] Video prediction usually relies on deep learning and artificial intelligence technologies, especially convolutional neural networks (CNNs), recurrent neural networks (RNNs) and their variants (such as LSTM, GRU, Transformer, etc.). The above models can learn patterns and rules from a large amount of video data and make accurate predictions. With the advancement of technology, the application scope of video prediction will become wider and wider, and the accuracy will continue to improve.

[0003] Diffusion models are a powerful generative model that was originally used for image generation tasks, but in recent years has also been applied to areas such as video prediction. However, videos are not just a simple combination of multiple static images. More importantly, there is temporal coherence and dynamic changes between frames. How to ensure a natural and smooth transition between generated video frames is a difficult problem for diffusion models. Summary of the invention

[0004] The embodiments of the present invention solve the technical problem in the prior art that the video energy obtained by prediction based on the diffusion model has low temporal consistency by providing a video generation method, device and equipment based on the diffusion model, and achieve the technical effect that the video energy obtained by prediction based on the trained diffusion model can maintain temporal consistency.

[0005] In a first aspect, the present invention provides a method for generating a video based on a diffusion model, the method comprising:

[0006] Extract frames from the video to be processed at a preset step length to obtain a number of extracted frame images;

[0007] According to the time sequence of the plurality of frame-drawing pictures and the preset construction quantity, a plurality of picture groups are constructed, wherein the preset construction quantity is at least 3 pictures, and each picture group, except the first picture group, sequentially includes the first frame-drawing picture and the last frame-drawing picture in the previous picture group;

[0008] Map the picture group in the initial data space to the latent space, where each frame in the picture group corresponds to a latent variable;

[0009] Constructing a diffusion model to be trained, wherein the input of the diffusion model to be trained is a picture group, and the output of the diffusion model to be trained is the predicted noise, the predicted picture group difference and the predicted picture difference;

[0010] According to the preset noise adding rules, add noise to each latent variable in the image group;

[0011] Inputting the picture groups into the diffusion model to be trained, obtaining a number of predicted noises and a number of predicted picture differences corresponding to each picture group, and a number of predicted picture group differences corresponding to the picture groups;

[0012] According to a number of predicted noises, a number of predicted picture differences and a number of predicted picture group differences, the neural network parameters of the diffusion model to be trained are adjusted, and when the preset training requirements are met, the diffusion model to be trained is used as the target diffusion model.

[0013] Further, according to the plurality of predicted noises, the plurality of predicted picture differences and the plurality of predicted picture group differences, the neural network parameters of the diffusion model to be trained are adjusted, including:

[0014] According to the predicted noise corresponding to each latent variable in each picture group and the actual added noise, the predicted noise difference corresponding to each latent variable is obtained;

[0015] According to the predicted picture difference between any two latent variables in the picture group and the actual picture difference between the two latent variables, a first predicted difference corresponding to the two latent variables is obtained;

[0016] Obtaining a second predicted difference corresponding to the two latent variables according to a predicted picture group difference between any two picture groups and an actual picture group difference between the two picture groups;

[0017] A neural network parameter of the diffusion model to be trained is adjusted according to the plurality of predicted noise differences, the plurality of first predicted differences, and the plurality of second predicted differences.

[0018] Furthermore, the picture group in the initial data space is mapped to the latent space, wherein each extracted frame picture in the picture group corresponds to a latent variable, including:

[0019] Encode each frame of the picture in the picture group to extract language features and spatial position features;

[0020] The image group in the initial data space is mapped to the latent space, and the latent variables corresponding to each extracted image are obtained according to the language features and spatial position features of each extracted image.

[0021] Furthermore, the loss function of the diffusion model to be trained includes:

[0022]

[0023] in, is the total difference, To predict the noise difference, is the first prediction difference, is the second prediction difference, and a and b are both preset weights.

[0024] Further, regarding the second prediction difference, it includes:

[0025]

[0026] in, is the expected value of the pixel difference, is the predicted pixel difference between the nth picture group and the mth picture group, is the actual pixel difference between the nth picture group and the mth picture group, is the expected value of the optical flow difference, is the predicted optical flow difference between the nth picture group and the mth picture group, is the actual optical flow difference between the nth picture group and the mth picture group, is the expected value of the cycle difference, is the prediction loop difference from the mth group of pictures back to the nth group of pictures, is the actual loop difference from the mth group of pictures back to the nth group, , as well as All are preset weights.

[0027] Furthermore, regarding the prediction noise difference, including

[0028]

[0029] in, To predict the expected value of the noise, is the difference between the predicted noise of the latent variable q and the predicted noise of the actual added noise in the nth picture group.

[0030] Furthermore, the order of extracting frames in the picture group includes:

[0031] Except for the first picture group, the first frame-drawing picture in the remaining picture groups is the first frame-drawing picture in the previous picture group of the picture group;

[0032] The second frame-drawing picture of the picture group is the last frame-drawing picture in the previous picture group of the picture group;

[0033] The third frame-drawing picture of the picture group is a frame-drawing picture arranged after the second frame-drawing picture of the picture group in the frame-drawing order.

[0034] Furthermore, the method further comprises:

[0035] Output the denoised latent variables and obtain the first video in the latent space in chronological order;

[0036] The first video is mapped back to the original data space, and the first video is used as the target video.

[0037] In a second aspect, the present invention provides a video generation device based on a diffusion model, the device comprising:

[0038] A frame extraction module is used to extract frames of the video to be processed with a preset step size to obtain a number of frame extraction pictures;

[0039] A grouping module, used to construct a plurality of picture groups according to the time sequence of the plurality of frame-drawing pictures and a preset construction quantity, wherein the preset construction quantity is at least 3 pictures, and each picture group, except the first picture group, sequentially includes the first frame-drawing picture and the last frame-drawing picture in the previous picture group;

[0040] A mapping module, used to map the picture group in the initial data space to the latent space, wherein each extracted frame picture in the picture group corresponds to a latent variable;

[0041] A construction module is used to construct a diffusion model to be trained, wherein the input of the diffusion model to be trained is a picture group, and the output of the diffusion model to be trained is a predicted noise, a predicted picture group difference, and a predicted picture difference;

[0042] An adding module, used to add noise to each latent variable in the image group according to a preset noise adding rule;

[0043] A difference acquisition module is used to input the picture group into the diffusion model to be trained, and obtain a number of predicted noises and a number of predicted picture differences corresponding to each picture group, as well as predicted picture group differences corresponding to a number of picture groups;

[0044] The training adjustment module is used to adjust the neural network parameters of the diffusion model to be trained according to a number of predicted noises, a number of predicted picture differences and a number of predicted picture group differences, and when the preset training requirements are met, the diffusion model to be trained is used as the target diffusion model.

[0045] In a third aspect, the present invention provides an electronic device, comprising:

[0046] processor;

[0047] a memory for storing processor-executable instructions;

[0048] The processor is configured to execute to implement the video generation method based on the diffusion model as provided in the first aspect.

[0049] One or more technical solutions provided in the embodiments of the present invention have at least the following technical effects or advantages:

[0050] The present invention provides a video generation method based on a diffusion model, the method comprising:

[0051] The video to be processed is framed with a preset step length to obtain a number of framed pictures; a number of picture groups are constructed according to the time sequence of the number of framed pictures and the preset construction quantity, wherein the preset construction quantity is at least 3 pictures, and each picture group, except the first picture group, sequentially contains the first framed picture and the last framed picture in the previous picture group; the picture group in the initial data space is mapped to the latent space, wherein each framed picture in the picture group corresponds to a latent variable; a diffusion model to be trained is constructed, the input of the diffusion model to be trained is the picture group, and the output of the diffusion model to be trained is the predicted noise, the predicted picture group difference and the predicted picture difference; according to the preset noise adding rule, noise is added to each latent variable in the picture group; the picture group is input into the diffusion model to be trained to obtain a number of predicted noises and a number of predicted picture differences corresponding to each picture group, as well as the predicted picture group differences corresponding to the number of picture groups; according to the number of predicted noises, the number of predicted picture differences and the number of predicted picture group differences, the neural network parameters of the diffusion model to be trained are adjusted, and when the preset training requirements are met, the diffusion model to be trained is used as the target diffusion model. The first frame-drawing picture of each picture group of the present invention is the first frame-drawing picture of the previous picture group, which ensures that the starting point of each picture group is consistent with the starting point of the previous picture group, and enhances the temporal coherence; the second frame-drawing picture of each picture group is the last frame-drawing picture of the previous picture group, which further ensures the continuity between the starting point and the end point of each picture group, so that the diffusion model to be trained can better capture the transition between frames, and by grouping in the above manner, the changes between adjacent picture groups are smoother, and mutations and discontinuities are reduced. Since the changes between adjacent picture groups are smoother, the diffusion model to be trained can converge faster and generalize better during the training process, reducing the risk of overfitting. Furthermore, in order to make the trained diffusion model have natural dynamics in both time and space, grouping in the above manner can enable the diffusion model to be trained to better learn how to retain important features and details during the generation process, thereby improving the quality of the generated video and making the generated video more coherent and smooth in time.

[0052] The present invention encodes each frame-sampling picture in the picture group, extracts language features and spatial position features, and maps the language features and spatial position features to the latent space, so that high-dimensional data can be converted into low-dimensional data. Moreover, based on the grouping of the frame-sampling pictures, the latent variables not only retain the language features and spatial position features, but also retain the order of each picture in the picture group and the order of the picture group. In other words, through the grouping-mapping process, the computational complexity is reduced and the computational efficiency is improved while retaining the temporal consistency.

[0053] If the present invention is adjusted according to [method 1], it can increase the calculation samples under the same input conditions to enhance the training effect of the diffusion model to be trained, so as to achieve the purpose of making the diffusion model obtained through training achieve consistency; if it is adjusted according to [method 2], it will help the diffusion model to be trained to better understand the time relationship between the picture groups and the time relationship between the frame pictures, which is more conducive to completing the training of time consistency, so that the video predicted by the diffusion model after training maintains time consistency.

[0054] When determining the difference between picture groups, the present invention not only considers the pixel difference, but also the total optical flow difference and the total cycle difference. Compared with only considering the difference between pictures, the present invention performs cycle comparison, pixel comparison and optical flow comparison on the picture group, so that the predicted video has better continuity and better temporal consistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0056] Figure 1 A schematic diagram of the process of the video generation method based on the diffusion model provided by the present invention;

[0057] Figure 2 A schematic diagram of the structure of a video generation device based on a diffusion model provided by the present invention;

[0058] Figure 3 The present invention provides a schematic structural diagram of an electronic device. DETAILED DESCRIPTION

[0059] The embodiment of the present invention solves the technical problem in the prior art that the temporal consistency of the video energy predicted based on the diffusion model is not high by providing a video generation method based on the diffusion model.

[0060] The technical solution of the embodiment of the present invention is to solve the above technical problems, and the overall idea is as follows:

[0061] A video generation method based on a diffusion model, the method comprising: extracting frames of a video to be processed with a preset step length to obtain a plurality of extracted frame images; constructing a plurality of picture groups according to the time sequence of the plurality of extracted frame images and a preset construction quantity, wherein the preset construction quantity is at least 3, and each picture group, except the first picture group, sequentially includes the first extracted frame image and the last extracted frame image in the previous picture group; mapping the picture group in the initial data space to the latent space, wherein each extracted frame image in the picture group corresponds to a latent variable; constructing a diffusion model to be trained, wherein the input of the diffusion model to be trained is the picture group, The outputs of the diffusion model to be trained are predicted noise, predicted picture group differences and predicted picture differences; according to the preset noise adding rule, noise is added to each latent variable in the picture group; the picture group is input into the diffusion model to be trained to obtain a number of predicted noises and a number of predicted picture differences corresponding to each picture group, as well as predicted picture group differences corresponding to a number of picture groups; according to the number of predicted noises, the number of predicted picture differences and the number of predicted picture group differences, the neural network parameters of the diffusion model to be trained are adjusted, and when the preset training requirements are met, the diffusion model to be trained is used as the target diffusion model.

[0062] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0063] First of all, the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0064] The present invention provides Figure 1 The video generation method based on the diffusion model shown includes steps S11-S17:

[0065] Regarding step S11, the video to be processed is frame-sampled at a preset step length to obtain a plurality of frame-sampled images.

[0066] Extracting frames from the video to be processed at a preset step size refers to extracting image frames from the video to be processed at a certain interval (i.e., the preset step size) to obtain a number of extracted frame images.

[0067] The amount of video data to be processed is usually large. Directly processing all frames may result in a waste of computing resources. Frame extraction can be used to reduce the amount of data to be processed and improve processing efficiency.

[0068] Furthermore, in order to better complete the training of the diffusion model, the video to be processed can be a video with uniform changes, for example, the real color and character movements of the video to be processed are all uniformly changed. It can be understood that a video to be processed with uniform changes means that the changes between adjacent frames are relatively smooth and continuous, making it easier for the diffusion model to be trained to capture and learn temporal dependencies, thereby reducing the complexity of modeling, and the difference between frames in the video to be processed with uniform changes is small, so the diffusion model to be trained can converge faster, improving training efficiency.

[0069] Furthermore, in order to improve the training quality of the diffusion model to be trained, different types of videos to be processed may be screened, and steps S11 to S17 may be repeatedly performed to repeatedly train the diffusion model.

[0070] Regarding step S12, several picture groups are constructed according to the time sequence of several frame-dropped pictures and the preset construction quantity, wherein the preset construction quantity is at least 3 pictures, and except for the first picture group, each picture group sequentially contains the first frame-dropped picture and the last frame-dropped picture in the previous picture group.

[0071] Specifically, the preset construction quantity refers to the preset number of extracted frame images in each image group. The preset construction quantity is at least 3 images. The specific preset construction quantity can be determined according to actual conditions.

[0072] The order of the extracted pictures in the picture group includes: except the first picture group, the first extracted picture in the remaining picture groups is the first extracted picture in the previous picture group of the picture group; the second extracted picture of the picture group is the last extracted picture in the previous picture group of the picture group; the third extracted picture of the picture group is the extracted picture arranged after the second extracted picture of the picture group according to the extraction order.

[0073] For example, after extracting frames from the video A to be processed, 1000 extracted frame images are obtained. The first 6 images of the 1000 extracted frame images are used as an example, where the preset construction quantity is 3, and the extracted frame images are recorded as 1, 2, 3, 4, 5, and 6 in chronological order. In chronological order, the first image group is (1, 2, 3), and each subsequent image group is: the second image group is (1, 3, 4), the third image group is (1, 4, 5), and the third image group is (1, 5, 6).

[0074] For example, the preset construction quantity is 3, and the extracted frame pictures are recorded as 1, 2, 3, 4, 5, 6, 7, and 8 in chronological order. In chronological order, the first picture group is (1, 2, 3, 4), and each subsequent picture group is: the second picture group is (1, 4, 5, 6), and the third picture group is (1, 6, 7, 8).

[0075] It can be understood that, according to the method provided above in the present invention, except for the first picture group, the first frame-drawing picture of each picture group is the first frame-drawing picture of the previous picture group, ensuring that the starting point of each picture group is consistent with the starting point of the previous picture group, enhancing the temporal coherence, and the second frame-drawing picture of each picture group is the last frame-drawing picture of the previous picture group, further ensuring the continuity between the starting point and the end point of each picture group, so that the diffusion model to be trained can better capture the transition between frames, and by grouping in the above manner, the changes between adjacent picture groups are smoother, and mutations and discontinuities are reduced. Since the changes between adjacent picture groups are smoother, the diffusion model to be trained can converge faster and generalize better during training, reducing the risk of overfitting.

[0076] Furthermore, in order to make the trained diffusion model have natural dynamics in both time and space, grouping in the above manner can enable the diffusion model to be trained to better learn how to retain important features and details during the generation process, thereby improving the quality of the generated video and making the generated video more coherent and smooth in time.

[0077] It can also be understood that the more preset build numbers there are, the better the above-mentioned effect can be achieved, but if the preset build number is too large, it will increase the training burden and make the training time too long. In order to balance the training effect and training time, the preset build number can be 4.

[0078] Regarding step S13, the picture group in the initial data space is mapped to the latent space, wherein each extracted frame picture in the picture group corresponds to a latent variable.

[0079] The initial data space refers to the space where the frame-extracted images are stored after the video to be processed is extracted. Each image group is also in the initial data space. Each frame-extracted image in the image group is high-dimensional data. The latent space is a low-dimensional space, in which each point (hidden variable) represents a frame-extracted image in the initial data space.

[0080] Specifically, each extracted frame picture in the picture group can be encoded to extract language features and spatial position features; the picture group in the initial data space is mapped to the latent space, and the latent variables corresponding to each extracted frame picture are obtained according to the language features and spatial position features of each extracted frame picture.

[0081] Language features provide high-level semantic information about the content of the image, helping to understand the objects, actions, and scenes in the image (can be implemented based on models such as BERT, RoBERTa, Transformer, etc.); spatial position features provide information about the location and layout of objects in the image, helping to understand the structure of the image (can be implemented based on the Transformer model).

[0082] The encoder can be used to map the language features and spatial position features to the latent space. By encoding each frame-by-frame picture in the picture group, extracting the language features and spatial position features, and mapping the language features and spatial position features to the latent space, high-dimensional data can be converted into low-dimensional data. Based on the grouping of the above-mentioned frame-by-frame pictures, the latent variables not only retain the language features and spatial position features, but also retain the order of each picture in the picture group and the order of the picture group. In other words, through the grouping-mapping process, the computational complexity is reduced and the computational efficiency is improved while retaining the temporal consistency.

[0083] Regarding step S14, a diffusion model to be trained is constructed, the input of the diffusion model to be trained is a picture group, and the output of the diffusion model to be trained is the predicted noise, the predicted picture group difference and the predicted picture difference.

[0084] It should be noted that the function of the traditional diffusion model is mainly to predict noise. In order for the diffusion model to realize video prediction, the input layer and output layer of the diffusion model to be trained can be adaptively modified to meet the purpose of video prediction. For example, the input of the traditional diffusion model is an image of H×W×C dimensions, where H×W is width and height, and C is the number of channels. In the present invention, in order to realize image difference prediction, the input can be set to H×W×(C1+C2), where C1 is the number of channels of the first latent variable, C2 is the number of channels of another latent variable, and the output is the predicted picture difference. Both the predicted picture group difference and the predicted noise can make adaptive modifications to the input and output, which will not be repeated here.

[0085] Regarding step S15, noise is added to each latent variable in the picture group according to a preset noise adding rule.

[0086] After constructing the diffusion model to be trained, noise can be added to each latent variable according to the preset noise adding rule. Specifically, the preset noise adding rule can be determined according to the actual situation. The present invention provides a preset noise adding rule, including: adding Gaussian noise to each latent variable in the picture group multiple times, and the intensity of the Gaussian noise added each time is the same. Noise can also be added in other ways, which will not be elaborated here.

[0087] Regarding step S16, the picture groups are input into the diffusion model to be trained to obtain a number of predicted noises and a number of predicted picture differences corresponding to each picture group, as well as predicted picture group differences corresponding to a number of picture groups.

[0088] It can be understood that each picture group contains at least 3 latent variables, so each time noise prediction is performed, the number of predicted noises in each group is the same as the number of latent variables in each picture group.

[0089] When predicting differences, multiple modes can be combined:

[0090] {Predict Image Difference}

[0091] A predicted picture difference can be predicted according to each two adjacent latent variables in the same picture group.

[0092] For example, the picture group is 1, 4, 5, 6, that is, (1, 4) is a predicted picture difference, (4, 5) is a predicted picture difference, and (5, 6) is a predicted picture difference.

[0093] Predicting according to the order in the picture group is more helpful for the diffusion model to be trained to understand the time order, so that the diffusion model obtained by training can achieve consistency.

[0094] It is also possible to predict a predicted picture difference according to any two latent variables in the same picture group.

[0095] For example, the picture groups are 1, 4, 5, and 6, that is, the combinations can be (1, 4), (4, 5), (5, 6), (1, 5), (1, 6), and (4, 6), for a total of 6 combinations, corresponding to 6 predicted picture differences.

[0096] By training the diffusion model to be trained in this way, calculation samples can be increased under the same input conditions to enhance the training effect of the diffusion model to be trained, thereby achieving the purpose of achieving consistency of the diffusion model obtained through training.

[0097] {Predicted image group differences}

[0098] Prediction can be performed on two adjacent picture groups in sequence to obtain a predicted picture group difference. Or,

[0099] It is also possible to perform prediction based on any two picture groups to obtain a predicted picture group difference.

[0100] The effects of "predicting according to two adjacent picture groups in sequence to obtain a predicted picture group difference" and "predicting according to each two adjacent latent variables in the same picture group to obtain a predicted picture difference" are similar and will not be elaborated here.

[0101] The effects of "predicting according to any two image groups to obtain a predicted image group difference" and "predicting according to any two latent variables in the same image group to obtain a predicted image difference" are similar and will not be elaborated here.

[0102] In addition, when using the uniformly transformed video to be trained, under the premise of ensuring the temporal consistency of each group of pictures, the difference between each group of pictures must be a linear difference. When using the diffusion model to be trained for prediction, not only should the linear changes between pictures be satisfied as much as possible, but also the linear changes can be satisfied at the macro level of the picture group. In other words, from a visual point of view, the predicted video should have coherence.

[0103] Regarding step S17, the neural network parameters of the diffusion model to be trained are adjusted according to the plurality of predicted noises, the plurality of predicted picture differences and the plurality of predicted picture group differences, and when the preset training requirements are met, the diffusion model to be trained is used as the target diffusion model.

[0104] Specifically, it can include method 1 or 2:

[0105] [Method 1]:

[0106] According to the predicted noise corresponding to each latent variable in each picture group and the actual added noise, the predicted noise difference corresponding to each latent variable is obtained; according to the predicted picture difference between any two latent variables in the picture group and the actual picture difference between the two latent variables, the first predicted difference corresponding to the two latent variables is obtained; according to the predicted picture group difference between any two picture groups and the actual picture group difference between the two picture groups, the second predicted difference corresponding to the two latent variables is obtained; according to a number of predicted noise differences, a number of first predicted differences and a number of second predicted differences, the neural network parameters of the diffusion model to be trained are adjusted.

[0107] [Method 2]:

[0108] According to the predicted noise corresponding to each latent variable in each picture group and the actual added noise, the predicted noise difference corresponding to each latent variable is obtained; according to the predicted picture difference between two adjacent latent variables in the picture group and the actual picture difference between the two latent variables, the first predicted difference corresponding to the two latent variables is obtained; according to the predicted picture group difference between two adjacent picture groups and the actual picture group difference between the two picture groups, the second predicted difference corresponding to the two latent variables is obtained; according to a number of predicted noise differences, a number of first predicted differences and a number of second predicted differences, the neural network parameters of the diffusion model to be trained are adjusted.

[0109] Unified description of [Method 1] and [Method 2]:

[0110] The actual added noise of each latent variable each time is a known quantity. The diffusion model can predict the predicted noise of each latent variable each time it is added according to the number of additions, and obtain a predicted noise difference based on the actual added noise each time and the corresponding predicted noise, thereby obtaining several predicted noise differences.

[0111] The image difference may include pixel difference and optical flow difference. The pixel difference in the image difference refers to the pixel difference between two latent variables, and the optical flow difference in the image difference refers to the difference in the optical flow field between two latent variables.

[0112] The picture group difference may include pixel difference, optical flow difference and cycle difference. The pixel difference in the picture group difference may refer to the sum of the pixel differences of each latent variable in the two picture groups. The optical flow difference in the picture group difference refers to the sum of the differences in the optical flow fields of two frame-sampling pictures at each same position in the two picture groups. For example, the picture groups (1, 2, 3) and (1, 3, 4), (1.1) is the same position, (2.3) is the same position, and (3.4) is the same position.

[0113] The cyclic difference in the GOP difference refers to a cyclic difference that circulates from one of the two GOPs back to the other GOP.

[0114] It should be noted that if the adjustment is performed according to [Method 1], the calculation samples can be increased under the same input conditions to enhance the training effect of the diffusion model to be trained, so as to achieve the purpose of achieving consistency of the diffusion model obtained through training.

[0115] If the adjustment is performed according to [Method 2], it will help the diffusion model to be trained to better understand the temporal relationship between the image groups and the temporal relationship between the extracted frames, which is more conducive to completing the training of temporal consistency, so that the video predicted by the trained diffusion model maintains temporal consistency.

[0116] It should be noted that when training is performed according to [Method 1], more samples are calculated, and because the time sequence of each frame-drawing picture is written when encoding each frame-drawing picture in step S13, the diffusion model obtained by training can achieve consistency, which is more accurate than [Method 2].

[0117] When training according to [Method 1], since the training is performed directly in chronological order, it helps the diffusion model to understand the chronological order between the frame images more quickly, and the training speed is faster, and the diffusion model can be obtained faster.

[0118] The present invention provides a loss function of a diffusion model to be trained, and the loss function of the diffusion model to be trained is used to evaluate the difference between the model prediction value and the actual value, and train the diffusion model according to the difference, including:

[0119]

[0120] in, is the total difference, To predict the noise difference, is the first prediction difference, is the second prediction difference, a and b are both preset weights, and the preset weight a and the preset weight b can be determined according to actual conditions, which will not be elaborated here.

[0121] Regarding the second prediction difference, it includes:

[0122]

[0123] in, is the expected value of the pixel difference, is the predicted pixel difference between the nth picture group and the mth picture group, is the actual pixel difference between the nth picture group and the mth picture group, is the expected value of the optical flow difference, is the predicted optical flow difference between the nth picture group and the mth picture group, is the actual optical flow difference between the nth picture group and the mth picture group, is the expected value of the cycle difference, is the prediction loop difference from the mth group of pictures back to the nth group of pictures, is the actual loop difference from the mth group of pictures back to the nth group, , as well as are all preset weights. Both represent norms.

[0124] The cyclic difference refers to the graphic deformation error generated in the process of transforming m picture groups into the nth picture group and then transforming the nth picture group back to the mth picture group. The pictures at each position in the picture group correspond one to one. Please refer to the above description for the corresponding rules.

[0125] When determining the difference between picture groups, the present invention not only considers the pixel difference, but also the total optical flow difference and the total cycle difference. Compared with only considering the difference between pictures, the present invention performs cycle comparison, pixel comparison and optical flow comparison on the picture group, so that the predicted video has better continuity and better temporal consistency.

[0126] The loss function for predicting noise differences includes

[0127]

[0128] in, To predict the expected value of the noise, is the difference between the predicted noise of the latent variable q and the predicted noise of the actual added noise in the nth picture group.

[0129] For the loss function of image difference, please refer to the loss function of image difference mentioned above. The two are similar and will not be described here.

[0130] Neural network parameters may include weight matrices, bias terms or other parameters, and preset training requirements may be meeting a maximum number of iterations or meeting accuracy requirements, which will not be elaborated here.

[0131] After the video is predicted, the denoised latent variables can be output, and the first video can be obtained in the latent space in chronological order (it is necessary to remove the first latent variables and the second latent variables of the remaining picture groups except the first picture group, and combine them in order to obtain the first video to achieve temporal consistency); the first video is mapped back to the initial data space, and the first video is used as the target video.

[0132] By reducing and increasing the dimensionality of the sampled frames in the initial data space, the training efficiency can be improved and a predicted video that can be used for viewing can be obtained.

[0133] In summary, the present invention provides a video generation method based on a diffusion model, the method comprising:

[0134] The video to be processed is framed with a preset step length to obtain a number of framed pictures; a number of picture groups are constructed according to the time sequence of the number of framed pictures and the preset construction quantity, wherein the preset construction quantity is at least 3 pictures, and each picture group, except the first picture group, sequentially contains the first framed picture and the last framed picture in the previous picture group; the picture group in the initial data space is mapped to the latent space, wherein each framed picture in the picture group corresponds to a latent variable; a diffusion model to be trained is constructed, the input of the diffusion model to be trained is the picture group, and the output of the diffusion model to be trained is the predicted noise, the predicted picture group difference and the predicted picture difference; according to the preset noise adding rule, noise is added to each latent variable in the picture group; the picture group is input into the diffusion model to be trained to obtain a number of predicted noises and a number of predicted picture differences corresponding to each picture group, as well as the predicted picture group differences corresponding to the number of picture groups; according to the number of predicted noises, the number of predicted picture differences and the number of predicted picture group differences, the neural network parameters of the diffusion model to be trained are adjusted, and when the preset training requirements are met, the diffusion model to be trained is used as the target diffusion model. The first frame-drawing picture of each picture group of the present invention is the first frame-drawing picture of the previous picture group, which ensures that the starting point of each picture group is consistent with the starting point of the previous picture group, and enhances the temporal coherence; the second frame-drawing picture of each picture group is the last frame-drawing picture of the previous picture group, which further ensures the continuity between the starting point and the end point of each picture group, so that the diffusion model to be trained can better capture the transition between frames, and by grouping in the above manner, the changes between adjacent picture groups are smoother, and mutations and discontinuities are reduced. Since the changes between adjacent picture groups are smoother, the diffusion model to be trained can converge faster and generalize better during the training process, reducing the risk of overfitting. Furthermore, in order to make the trained diffusion model have natural dynamics in both time and space, grouping in the above manner can enable the diffusion model to be trained to better learn how to retain important features and details during the generation process, thereby improving the quality of the generated video and making the generated video more coherent and smooth in time.

[0135] The present invention encodes each frame-sampling picture in the picture group, extracts language features and spatial position features, and maps the language features and spatial position features to the latent space, so that high-dimensional data can be converted into low-dimensional data. Moreover, based on the grouping of the frame-sampling pictures, the latent variables not only retain the language features and spatial position features, but also retain the order of each picture in the picture group and the order of the picture group. In other words, through the grouping-mapping process, the computational complexity is reduced and the computational efficiency is improved while retaining the temporal consistency.

[0136] If the present invention is adjusted according to [method 1], it can increase the calculation samples under the same input conditions to enhance the training effect of the diffusion model to be trained, so as to achieve the purpose of making the diffusion model obtained through training achieve consistency; if it is adjusted according to [method 2], it will help the diffusion model to be trained to better understand the time relationship between the picture groups and the time relationship between the frame pictures, which is more conducive to completing the training of time consistency, so that the video predicted by the diffusion model after training maintains time consistency.

[0137] When determining the difference between picture groups, the present invention not only considers the pixel difference, but also the total optical flow difference and the total cycle difference. Compared with only considering the difference between pictures, the present invention performs cycle comparison, pixel comparison and optical flow comparison on the picture group, so that the predicted video has better continuity and better temporal consistency.

[0138] Based on the same inventive concept, the present invention provides Figure 2 The video generation device based on the diffusion model shown includes:

[0139] The frame extraction module 21 is used to extract frames of the video to be processed with a preset step length to obtain a plurality of frame extraction pictures;

[0140] A grouping module 22 is used to construct a plurality of picture groups according to the time sequence of the plurality of frame-drawing pictures and a preset construction quantity, wherein the preset construction quantity is at least 3 pictures, and each picture group, except the first picture group, sequentially includes the first frame-drawing picture and the last frame-drawing picture in the previous picture group;

[0141] A mapping module 23, used for mapping the picture group in the initial data space to the latent space, wherein each extracted frame picture in the picture group corresponds to a latent variable;

[0142] A construction module 24 is used to construct a diffusion model to be trained, wherein the input of the diffusion model to be trained is a picture group, and the output of the diffusion model to be trained is a predicted noise, a predicted picture group difference, and a predicted picture difference;

[0143] An adding module 25, configured to add noise to each latent variable in the picture group according to a preset noise adding rule;

[0144] The difference acquisition module 26 is used to input the picture group into the diffusion model to be trained, and obtain a number of predicted noises and a number of predicted picture differences corresponding to each picture group, and a number of predicted picture group differences corresponding to the picture groups;

[0145] The training adjustment module 27 is used to adjust the neural network parameters of the diffusion model to be trained according to a number of predicted noises, a number of predicted picture differences and a number of predicted picture group differences, and when the preset training requirements are met, the diffusion model to be trained is used as the target diffusion model.

[0146] Based on the same inventive concept, the present invention also provides Figure 3 An electronic device as shown includes:

[0147] Processor 31;

[0148] A memory 32 for storing instructions executable by the processor 31;

[0149] The processor 31 is configured to execute to implement the video generation method based on the diffusion model as provided above.

[0150] Since the electronic device introduced in this embodiment is an electronic device used to implement the information processing method in the embodiment of the present invention, based on the information processing method introduced in the embodiment of the present invention, a person skilled in the art can understand the specific implementation of the electronic device of this embodiment and its various variations, so how the electronic device implements the method in the embodiment of the present invention is not described in detail here. As long as the electronic device used by a person skilled in the art to implement the information processing method in the embodiment of the present invention, it belongs to the scope of protection of the present invention.

[0151] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0152] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0153] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0154] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0155] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0156] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A video generation method based on a diffusion model, characterized in that: The method comprises: Extract frames from the video to be processed at a preset step length to obtain a number of extracted frame images; According to the time sequence of the plurality of frame-drawing pictures and the preset construction quantity, a plurality of picture groups are constructed, wherein the preset construction quantity is at least 3 pictures, and each picture group, except the first picture group, sequentially includes the first frame-drawing picture and the last frame-drawing picture in the previous picture group; Map the picture group in the initial data space to the latent space, where each frame in the picture group corresponds to a latent variable; Constructing a diffusion model to be trained, wherein the input of the diffusion model to be trained is a picture group, and the output of the diffusion model to be trained is a predicted noise, a predicted picture group difference, and a predicted picture difference; According to the preset noise adding rules, add noise to each latent variable in the image group; Inputting the picture groups into the diffusion model to be trained, obtaining a number of predicted noises and a number of predicted picture differences corresponding to each picture group, and a number of predicted picture group differences corresponding to the picture groups; According to a number of predicted noises, a number of predicted picture differences, and a number of predicted picture group differences, the neural network parameters of the diffusion model to be trained are adjusted, and when the preset training requirements are met, the diffusion model to be trained is used as the target diffusion model, including: according to the predicted noise corresponding to each latent variable in each picture group and the actual added noise, the predicted noise difference corresponding to each latent variable is obtained; according to the predicted picture difference between any two latent variables in the picture group and the actual picture difference between the two latent variables, the first predicted difference corresponding to the two latent variables is obtained; according to the predicted picture group difference between any two picture groups and the actual picture group difference between the two picture groups, the second predicted difference corresponding to the two latent variables is obtained; according to the predicted noise difference, the first predicted difference, and the second predicted difference, the neural network parameters of the diffusion model to be trained are adjusted; Among them, the loss function of the diffusion model to be trained includes: in, is the total difference, To predict the noise difference, is the first prediction difference, is the second prediction difference, and a and b are both preset weights.

2. The video generation method based on the diffusion model according to claim 1, characterized in that: The mapping of the picture group in the initial data space to the latent space, wherein each frame-drawing picture in the picture group corresponds to a latent variable, includes: Encode each frame of the picture in the picture group to extract language features and spatial position features; The image group in the initial data space is mapped to the latent space, and the latent variables corresponding to each extracted image are obtained according to the language features and spatial position features of each extracted image.

3. The video generation method based on diffusion model according to claim 1, characterized in that: Regarding the second prediction difference, it includes: in, is the expected value of the pixel difference, is the predicted pixel difference between the nth picture group and the mth picture group, is the actual pixel difference between the nth picture group and the mth picture group, is the expected value of the optical flow difference, is the predicted optical flow difference between the nth picture group and the mth picture group, is the actual optical flow difference between the nth picture group and the mth picture group, is the expected value of the cycle difference, is the prediction loop difference from the mth group of pictures back to the nth group of pictures, is the actual loop difference from the mth group of pictures back to the nth group, , as well as All are preset weights.

4. The video generation method based on diffusion model according to claim 1, characterized in that: Regarding the prediction noise differences, including in, To predict the expected value of the noise, is the difference between the predicted noise of the latent variable q and the predicted noise of the actual added noise in the nth picture group.

5. The video generation method based on diffusion model according to claim 1, characterized in that: The order of the extracted frames in the picture group includes: Except for the first picture group, the first frame-drawing picture in the remaining picture groups is the first frame-drawing picture in the previous picture group of the picture group; The second frame-drawing picture of the picture group is the last frame-drawing picture in the previous picture group of the picture group; The third frame-drawing picture of the picture group is a frame-drawing picture arranged after the second frame-drawing picture of the picture group in the frame-drawing order.

6. The video generation method based on diffusion model according to claim 1, characterized in that: The method further comprises: Output the denoised latent variables and obtain the first video in the latent space in chronological order; The first video is mapped back to the initial data space, and the first video is used as the target video.

7. A video generation device based on a diffusion model, characterized in that: The method for generating a video based on a diffusion model according to any one of claims 1 to 6, wherein the device comprises: A frame extraction module is used to extract frames of the video to be processed with a preset step size to obtain a number of frame extraction pictures; A grouping module, used to construct a plurality of picture groups according to the time sequence of the plurality of frame-drawing pictures and a preset construction quantity, wherein the preset construction quantity is at least 3 pictures, and each picture group, except the first picture group, sequentially includes the first frame-drawing picture and the last frame-drawing picture in the previous picture group; A mapping module, used to map the picture group in the initial data space to the latent space, wherein each extracted frame picture in the picture group corresponds to a latent variable; A construction module, used for constructing a diffusion model to be trained, wherein the input of the diffusion model to be trained is a picture group, and the output of the diffusion model to be trained is a predicted noise, a predicted picture group difference and a predicted picture difference; An adding module, used to add noise to each latent variable in the image group according to a preset noise adding rule; A difference acquisition module, used for inputting the picture group into the diffusion model to be trained, and obtaining a number of predicted noises and a number of predicted picture differences corresponding to each picture group, and a number of predicted picture group differences corresponding to the picture groups; The training adjustment module is used to adjust the neural network parameters of the diffusion model to be trained according to a number of predicted noises, a number of predicted picture differences and a number of predicted picture group differences, and when the preset training requirements are met, the diffusion model to be trained is used as the target diffusion model.

8. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute to implement the video generation method based on the diffusion model as described in any one of claims 1 to 6.

Images