Video generation method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to a video generation method and device, electronic equipment, a storage medium and a program product. The method comprises the following steps: acquiring a source frame sequence of a source video, and acquiring prompt information for the source video; the source frame sequence comprises a plurality of continuous video frames in the source video; for a video frame pair formed by adjacent video frames in the source frame sequence, predicting an optical flow field of the video frame pair; determining a plurality of pixel moving paths of the source frame sequence based on the optical flow field; the pixel moving path represents the position change of the same pixel on at least two continuous video frames in the source frame sequence; adding noise to each video frame in the source frame sequence to obtain a noise frame representation, and generating a noise frame representation sequence formed by the noise frame representation corresponding to each video frame in the source frame sequence; and iteratively predicting noise according to the noise frame representation sequence, the prompt information and the plurality of pixel moving paths, and iteratively denoising the noise frame representation sequence based on the predicted noise to generate a target video. By adopting the method, the visual consistency can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of video generation, and in particular to a video generation method, device, electronic device, storage medium and program product. Background Art

[0002] With the development of computer technology, AIGC (Artificial Intelligence Generated Content) technology has emerged. AIGC technology is a technical method based on artificial intelligence, which generates relevant content with appropriate generalization ability through learning and identifying existing data. Through training models and learning a large amount of data, AIGC technology can generate relevant content based on input conditions or guidance. AIGC technology is widely used in various fields. For example, it can be applied to the field of video generation. In the field of video generation, through artificial intelligence models, driven by prompt information such as text prompts and image prompts, the source video can be edited into a new video that matches the prompt information to generate a new video.

[0003] However, in the prior art, the generated videos have the problem of poor visual consistency. Summary of the invention

[0004] Based on this, it is necessary to provide a video generation method, device, electronic device, storage medium and program product that can improve visual consistency in response to the above technical problems.

[0005] In a first aspect, the present application provides a video generation method, comprising:

[0006] Acquire a source frame sequence of a source video, and acquire prompt information for the source video; the source frame sequence includes a plurality of continuous video frames in the source video;

[0007] For each video frame pair consisting of two adjacent video frames in the source frame sequence, predicting the optical flow field of the video frame pair;

[0008] Determine a plurality of pixel movement paths of the source frame sequence based on the optical flow field; the pixel movement paths represent position changes of the same pixel in at least two consecutive video frames in the source frame sequence;

[0009] Noising each video frame in the source frame sequence to obtain a noise frame representation, and generating a noise frame representation sequence consisting of the noise frame representation corresponding to each video frame in the source frame sequence;

[0010] According to the noise frame representation sequence, the prompt information and the multiple pixel movement paths, noise is iteratively predicted, and the noise frame representation sequence is iteratively denoised based on the predicted noise to generate a target video.

[0011] In a second aspect, the present application further provides a video generation device, including:

[0012] An acquisition module, used to acquire a source frame sequence of a source video and acquire prompt information for the source video; the source frame sequence includes a plurality of continuous video frames in the source video;

[0013] An optical flow prediction module, configured to predict an optical flow field of each video frame pair consisting of two adjacent video frames in the source frame sequence;

[0014] A path determination module, configured to determine a plurality of pixel movement paths of the source frame sequence based on the optical flow field; the pixel movement paths represent position changes of the same pixel in at least two consecutive video frames in the source frame sequence;

[0015] The video generation module is used to add noise to each video frame in the source frame sequence to obtain a noise frame representation, and generate a noise frame representation sequence consisting of the noise frame representation corresponding to each video frame in the source frame sequence; iteratively predict noise according to the noise frame representation sequence, the prompt information and the multiple pixel movement paths, and iteratively denoise the noise frame representation sequence based on the predicted noise to generate a target video.

[0016] In a third aspect, the present application further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes one or more programs, the following steps are implemented:

[0017] Acquire a source frame sequence of a source video, and acquire prompt information for the source video; the source frame sequence includes a plurality of continuous video frames in the source video;

[0018] For each video frame pair consisting of two adjacent video frames in the source frame sequence, predicting the optical flow field of the video frame pair;

[0019] Determine a plurality of pixel movement paths of the source frame sequence based on the optical flow field; the pixel movement paths represent position changes of the same pixel in at least two consecutive video frames in the source frame sequence;

[0020] Noising each video frame in the source frame sequence to obtain a noise frame representation, and generating a noise frame representation sequence consisting of the noise frame representation corresponding to each video frame in the source frame sequence;

[0021] According to the noise frame representation sequence, the prompt information and the multiple pixel movement paths, noise is iteratively predicted, and the noise frame representation sequence is iteratively denoised based on the predicted noise to generate a target video.

[0022] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0023] Acquire a source frame sequence of a source video, and acquire prompt information for the source video; the source frame sequence includes a plurality of continuous video frames in the source video;

[0024] For each video frame pair consisting of two adjacent video frames in the source frame sequence, predicting the optical flow field of the video frame pair;

[0025] Determine a plurality of pixel movement paths of the source frame sequence based on the optical flow field; the pixel movement paths represent position changes of the same pixel in at least two consecutive video frames in the source frame sequence;

[0026] Noising each video frame in the source frame sequence to obtain a noise frame representation, and generating a noise frame representation sequence consisting of the noise frame representation corresponding to each video frame in the source frame sequence;

[0027] According to the noise frame representation sequence, the prompt information and the multiple pixel movement paths, noise is iteratively predicted, and the noise frame representation sequence is iteratively denoised based on the predicted noise to generate a target video.

[0028] In a fifth aspect, the present application further provides a computer program product, including a computer program, which, when executed by a processor, implements the following steps:

[0029] Acquire a source frame sequence of a source video, and acquire prompt information for the source video; the source frame sequence includes a plurality of continuous video frames in the source video;

[0030] For each video frame pair consisting of two adjacent video frames in the source frame sequence, predicting the optical flow field of the video frame pair;

[0031] Determine a plurality of pixel movement paths of the source frame sequence based on the optical flow field; the pixel movement paths represent position changes of the same pixel in at least two consecutive video frames in the source frame sequence;

[0032] Noising each video frame in the source frame sequence to obtain a noise frame representation, and generating a noise frame representation sequence consisting of the noise frame representation corresponding to each video frame in the source frame sequence;

[0033] According to the noise frame representation sequence, the prompt information and the multiple pixel movement paths, noise is iteratively predicted, and the noise frame representation sequence is iteratively denoised based on the predicted noise to generate a target video.

[0034] The above-mentioned video generation method, device, electronic device, storage medium and program product predict the optical flow field for each video frame pair in the source frame sequence, and then determine multiple pixel movement paths of the source frame sequence based on the optical flow field, and add noise to each video frame in the source frame sequence to generate a noise frame representation sequence. Since the pixel movement paths in the multiple pixel movement paths of the source frame sequence represent the position change of the same pixel on at least two consecutive video frames in the source frame sequence, in the process of iteratively predicting noise based on the noise frame representation sequence, prompt information and multiple pixel movement paths, it is possible to pay attention to the position change of pixels on consecutive video frames, pay attention to the continuity between video frames of the source video, that is, pay attention to the characteristics of visual consistency. In this way, the noise frame representation sequence is iteratively denoised based on the predicted noise to generate a target video, thereby improving the visual consistency of the target video. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 is a flow chart of a video generation method in one embodiment;

[0037] Figure 2 A flowchart of a step of iteratively predicting noise at multiple denoising time steps through a noise prediction network according to a noise frame representation sequence, prompt information, and multiple pixel movement paths in one embodiment;

[0038] Figure 3 is a structural block diagram of a video generating device in one embodiment;

[0039] Figure 4 FIG. 4 is a diagram showing the internal structure of an electronic device in one embodiment. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0041] In one embodiment, Figure 1As shown, a video generation method is provided. This embodiment uses the method applied to an electronic device as an example. The electronic device can be a terminal or a server. It can be understood that the method can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. The terminal can be a personal computer, a laptop, a smart phone, a tablet computer or other. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. In this embodiment, the method includes the following steps:

[0042] Step 102, obtaining a source frame sequence of a source video and obtaining prompt information for the source video; the source frame sequence includes a plurality of continuous video frames in the source video.

[0043] The source video is a video edited to generate a target video. The multiple consecutive video frames are multiple video frames arranged in sequence in the source video according to the playback order. The prompt information is the guidance information for reference in editing the source video to generate the target video. The prompt information can be text prompt information, image prompt information or other. For example, the text prompt information can be "change a kitten to a puppy", then during the video generation process, the image of a kitten in the source video can be changed to the image of a puppy to generate the target video.

[0044] Exemplarily, the electronic device may obtain the source video and the prompt information for the source video in response to the video generation trigger event, and extract the source frame sequence of the source video from the source video.

[0045] The video generation trigger event is an event that triggers the generation of a video. The video generation trigger event may be an automatic trigger event, for example, a timed trigger after a timed generation task for a source video is created. The video generation trigger event may be a manual trigger operation, for example, a trigger operation on a video generation control in a video generation software. The source video may be uploaded by a user or may be a locally stored source video. The prompt information may be input by a user or may be generated using a machine learning model, for example, may be random text generated using a large language model (LLM).

[0046] Step 104 : predicting the optical flow field of each video frame pair consisting of two adjacent video frames in the source frame sequence.

[0047] The optical flow field of the video frame pair characterizes the position change of the same pixel on the video frame pair. The video frame pair may include a first video frame and a second video frame. When predicting the optical flow field of the video frame pair, for each pixel in the first video frame, the position of the pixel in the second video frame is predicted. It can be understood that the brightness of the same pixel remains unchanged in the video frame pair, and the brightness of the pixel can be the grayscale value of the pixel. The optical flow field of the video frame pair can be specifically represented by a displacement vector of the pixel, and the displacement vector can include the displacement direction and displacement magnitude of the pixel position.

[0048] Exemplarily, the electronic device may predict the optical flow field of each video frame pair consisting of two adjacent video frames in the source frame sequence by using an optical flow algorithm. The optical flow algorithm may be a Lucas-Kanade algorithm (an optical flow estimation algorithm proposed by Bruce D. Lucas and Takeo Kanade), a FlowNet algorithm (an optical flow prediction algorithm based on a convolutional neural network, with different versions including FlowNetS, FlowNetC, FlowNet2, etc.), a RAFT algorithm (Recurrent All-Pairs Field Transforms, a recursive full-pairs field transformation optical flow calculation model proposed by Zachary Teed and Jia Deng of Princeton University), or others.

[0049] Step 106: determine multiple pixel movement paths of the source frame sequence based on the optical flow field; the pixel movement paths represent the position change of the same pixel in at least two consecutive video frames in the source frame sequence.

[0050] The pixel movement path may be a sequence formed by the positions of the same pixel on at least two consecutive video frames in the source frame sequence. The position of the pixel on the video frame may be represented by coordinates. It is understandable that the video frame pair may include a first video frame and a second video frame, and the optical flow field of the video frame pair may be a displacement vector of the pixel of the first video frame in the video frame pair. After determining the position of each pixel in the first video frame, the position of each pixel in the first video frame in the second video frame may be determined through the optical flow field of the video frame pair. It is understandable that each position in the pixel movement path is the position of the pixel in different video frames, and a position in a video frame may be taken as a position in the pixel movement path.

[0051] At least two consecutive video frames are consecutive video frames containing the same pixel. It is understandable that the same pixel may appear in all video frames of the source frame sequence, or may only appear in a continuous portion of video frames of the source frame sequence. For example, the source frame sequence may include K video frames. As the video frames progress, the image of a portion of the screen area in the video frame may be replaced by a new image, that is, some pixels may disappear from the source frame sequence, and new pixels may appear from the middle video frame. Then, the partial video frames may be the first video frame to the kth video frame, or the kth video frame to the Kth video frame, or the number of video frames between the first video frame and the Kth video frame, which is less than K. K is a positive integer greater than 2, and k is any positive integer between 1 and K.

[0052] Exemplarily, the electronic device may start from the first video frame in the source frame sequence, and iteratively determine the position of each pixel in the source frame sequence on at least two consecutive video frames in the source frame sequence based on the optical flow field of each video frame pair; determine the sequence formed by the position of each pixel in the source frame sequence on at least two consecutive video frames as the pixel movement path of each pixel, and obtain multiple pixel movement paths of the source frame sequence.

[0053] In one embodiment, the electronic device can determine the position of each pixel in the first video frame in the source frame sequence, starting from the first video frame in the source frame sequence, iteratively determine the position of each pixel in the first video frame on at least two consecutive video frames based on the optical flow field of each video frame pair, and when a new pixel that does not exist in the first video frame and appears in a video frame after the first video frame appears, determine the position of the new pixel on at least two consecutive video frames; determine the sequence formed by the position of each pixel in the source frame sequence on at least two consecutive video frames as the pixel movement path of each pixel, and obtain multiple pixel movement paths of the source frame sequence.

[0054] In one embodiment, after determining multiple pixel movement paths of a source frame sequence based on an optical flow field, the multiple pixel movement paths of the source frame sequence can be downsampled using a preconfigured latent space size to obtain downsampled pixel movement paths, and the downsampled pixel movement paths are updated to the multiple pixel movement paths of the source frame sequence.

[0055] The latent space size is the size of the noise frame in the noise frame sequence input to the noise prediction network, for example, it can be 64*64, or 32*32. Downsampling multiple pixel movement paths of the source frame sequence can be to reduce the size of each video frame in the source frame sequence to the same as the latent space size, and then map the multiple pixel movement paths of the source frame sequence determined based on the optical flow field to the reduced-size source frame sequence. For example, the size of the video frame can be 1024*1024, the latent space size can be 64*64, and a pixel movement path of the source frame sequence determined based on the optical flow field can be , since 1024 is 16 times of 64, the pixel movement path after downsampling can be .in, This could be the position of the pixel in the first video frame in the source frame sequence, and so on. It may be the position of the pixel in the Kth video frame in the source frame sequence. After downsampling the multiple pixel movement paths, there may be repeated pixel movement paths, and the repeated pixel movement paths may be deduplicated.

[0056] Step 108 , adding noise to each video frame in the source frame sequence to obtain a noise frame representation, and generating a noise frame representation sequence consisting of the noise frame representation corresponding to each video frame in the source frame sequence.

[0057] The noise frame representation is an implicit representation of the noisy video frame. The implicit representation can also be called latent space representation.

[0058] Exemplarily, the electronic device may process each video frame in the source frame sequence into a source frame representation in the latent space, obtain a source frame representation sequence consisting of source frame representations corresponding to each video frame in the source frame sequence, add random noise to each source frame representation in the source frame representation sequence, obtain a noise frame representation, and generate a noise frame representation sequence consisting of noise frame representations corresponding to each video frame in the source frame sequence. The source frame representation is a data representation of the video frame in the latent space.

[0059] In one embodiment, the electronic device may process each video frame in the source frame sequence as a source frame representation in a latent space, and obtain a source frame representation sequence consisting of source frame representations corresponding to each video frame in the source frame sequence; perform iterative noise prediction for multiple noise adding time steps, and after predicting the noise of each noise adding time step, add noise to the source frame representation sequence output at the previous noise adding time step based on the noise predicted at the current noise adding time step, and obtain a source frame representation sequence output at the current noise adding time step; determine each source frame representation in the source frame representation sequence output at the last noise adding time step as a noise frame representation, and obtain a noise frame representation sequence consisting of noise frame representations corresponding to each video frame in the source frame sequence.

[0060] The noise adding time step is the time step of the noise adding process. To the noisy data In the process of adding noise, from the clear data To the data after adding noise , you can go through For example, the first noise-added time step can be Add noise to get , the second noise-adding time step can be Add noise to get , and so on, until we get , then the iterative noise addition is completed. It can be a positive integer greater than 1. For example, T can be 100, 1000 or others.

[0061] In the iterative prediction noise of multiple noise adding time steps, the multiple noise adding time steps can be each noise adding time step in T noise adding time steps; the multiple noise adding time steps can also be a part of T noise adding time steps, for example, T can be 100, and the multiple noise adding time steps can be the first and integer multiples of the 10th noise adding time step from the 1st noise adding time step to the 100th noise adding time step, so that the skipping prediction noise can be realized and the noise prediction efficiency can be improved. The previous noise adding time step is the previous one of the current noise adding time step in T noise adding time steps, for example, if the current noise adding time step is 10, the previous noise adding time step can be 9.

[0062] In one embodiment, the electronic device may iteratively denoise the source frame representation sequence based on the predicted noise and the preset denoising formula. The preset denoising formula may be a denoising formula of a preconfigured diffusion model, such as a denoising formula of a DDPM model or a DDIM model. For example, the denoising formula of the DDIM model may be written as formula (1):

[0063] Formula (1)

[0064] in, It can represent the noise data labeled t (which can be understood as the data of the tth time step) from the clear data labeled 0 (which can be understood as the data of the 0th time step) to the noise data labeled T (which can be understood as the data of the Tth time step). And so on. It can be understood as The noise frame output at the previous noise time step represents the sequence. , The amount of noise at each time step can be controlled by setting scheduler hyperparameters. The noise that can be predicted by the noise prediction network.

[0065] In one embodiment, the step of adding noise to the source frame representation sequence output at the previous noisy time step based on the noise predicted at the current noisy time step may include: when the current noisy time step is the first noisy time step, adding noise to the source frame representation sequence based on the noise predicted at the current noisy time step.

[0066] In one embodiment, the step of iteratively predicting noise for multiple noise-added time steps may include: the electronic device may iteratively predict noise for multiple noise-added time steps, and in each noise-added time step, randomly sample from a normal distribution to obtain the noise of the current noise-added time step.

[0067] In one embodiment, the step of iteratively predicting noise for multiple noise adding time steps may include: the electronic device may iteratively predict noise for multiple noise adding time steps through a noise prediction network based on a source frame representation sequence and multiple pixel movement paths.

[0068] Step 110, iteratively predicting noise according to the noise frame representation sequence, prompt information, and multiple pixel movement paths, iteratively denoising the noise frame representation sequence based on the predicted noise, and generating a target video.

[0069] The target video is a video obtained by editing the source video according to the prompt information. Specifically, the target video may be a video obtained by editing the video frames in the source video according to the prompt information.

[0070] Exemplarily, the electronic device may iteratively predict noise through a noise prediction network according to the noise frame representation sequence, prompt information, and multiple pixel movement paths, iteratively denoise the noise frame representation sequence based on the predicted noise, and generate a target video.

[0071] Among them, the noise prediction network is a network for predicting noise. The noise prediction network can be a network obtained by the UNet network based on the diffusion model. Specifically, the two-dimensional convolution of the UNet network of the diffusion model can be changed to a three-dimensional convolution, in which one axis indexes the video frame and the other two axes index the spatial height and width, so that the UNet network can process the video. For example, the 3x3 convolution is changed to a 1x3x3 convolution, the first axis indexes the video frame, and the second and third axes index the spatial height and width. The diffusion model can be a DDPM model (Denoising Diffusion Probabilistic Models, a generative model based on Markov chain), a DDIM model (Denoising Diffusion Implicit Models, which is an improved model of the DDPM model and a generative model based on non-Markov chain) or others.

[0072] In one embodiment, the electronic device may iteratively denoise the noise frame representation sequence based on the predicted noise and the preset denoising formula. The preset denoising formula may be a denoising formula of a pre-configured diffusion model, such as a denoising formula of a DDPM model or a DDIM model. For example, the denoising formula of the DDIM model may be written as formula (2):

[0073] Formula (2)

[0074] in, It can represent the noise data labeled T (which can be understood as the data of the Tth time step) to the clear data labeled 0 (which can be understood as the data of the 0th time step), and the noise data labeled t (which can be understood as the data of the tth time step). And so on. It can be understood as The noisy frames output at the previous denoising time step represent the sequence. , The amount of noise at each time step can be controlled by setting scheduler hyperparameters. The noise that can be predicted by the noise prediction network.

[0075] In one embodiment, the electronic device may iteratively predict noise through a noise prediction network according to the noise frame representation sequence, prompt information, and multiple pixel movement paths. After each iteration, the noise frame representation sequence outputted in the previous iteration is denoised based on the noise predicted in the current iteration to obtain the noise frame representation sequence outputted in the current iteration, and the target video is generated based on the noise frame representation sequence outputted in the last iteration. In the first iteration, after the noise is predicted, the noise frame representation sequence obtained in step 108 is denoised based on the noise predicted in the current iteration to obtain the noise frame representation sequence outputted in the current iteration.

[0076] In the above-mentioned video generation method, an optical flow field is predicted for each video frame pair in a source frame sequence, and then multiple pixel movement paths of the source frame sequence are determined based on the optical flow field. Each video frame in the source frame sequence is denoised to generate a noise frame representation sequence. Since the pixel movement paths in the multiple pixel movement paths of the source frame sequence represent the position change of the same pixel on at least two consecutive video frames in the source frame sequence, in the process of iteratively predicting noise based on the noise frame representation sequence, prompt information and multiple pixel movement paths, it is possible to pay attention to the position change of pixels on consecutive video frames and the continuity between video frames of the source video, that is, to pay attention to the feature of visual consistency. In this way, the noise frame representation sequence is iteratively denoised based on the predicted noise to generate a target video, thereby improving the visual consistency of the target video.

[0077] In an exemplary embodiment, step 110 may include: iteratively predicting noise for multiple denoising time steps through a noise prediction network according to a noise frame representation sequence, prompt information, and multiple pixel movement paths; after predicting the noise of each denoising time step, denoising the noise frame representation sequence output at the previous denoising time step based on the noise predicted at the current denoising time step to obtain a noise frame representation sequence output at the current denoising time step; and generating a target video based on the noise frame representation sequence output at the last denoising time step.

[0078] The denoising time step is the time step of the denoising process. To clear data It is understandable that the noise in the process is the original data, such as the source video; is the generated data, such as the target video. In the denoising process, the data after adding noise To clear data , you can go through denoising time steps, for example, the first denoising time step can be Denoising , the second denoising time step can be Denoising , and so on, until we get , then the iterative denoising is completed. It can be understood that in the denoising process, the skip-step denoising method of the DDIM model can also be used.

[0079] In this embodiment, according to the noise frame representation sequence, prompt information and multiple pixel movement paths, the noise is iteratively predicted for multiple denoising time steps through a noise prediction network. Moreover, after the noise of each denoising time step is predicted, the noise frame representation sequence output at the previous denoising time step is denoised based on the noise predicted at the current denoising time step to obtain the noise frame representation sequence output at the current denoising time step. By predicting the noise and denoising at multiple denoising time steps, more accurate denoising can be achieved, and a more accurate target video can be output.

[0080] In one embodiment, according to the noise frame representation sequence, prompt information and multiple pixel movement paths, a noise prediction network is used to iteratively predict noise for multiple denoising time steps; after the noise of each denoising time step is predicted, the noise frame representation sequence output at the previous denoising time step is denoised based on the noise predicted at the current denoising time step to obtain the noise frame representation sequence output at the current denoising time step, including:

[0081] In the first denoising time step, according to the noise frame representation sequence, prompt information and multiple pixel movement paths obtained in step 108, the noise is predicted through the noise prediction network, and the noise frame representation sequence is denoised based on the noise predicted in the current denoising time step and the preset denoising formula to obtain the noise frame representation sequence output in the current denoising time step; starting from the second denoising time step, in each denoising time step, according to the noise frame representation sequence, prompt information and multiple pixel movement paths output in the previous denoising time step, the noise is predicted through the noise prediction network, and based on the noise predicted in the current denoising time step and the preset denoising formula, the noise frame representation sequence output in the previous denoising time step is denoised to obtain the noise frame representation sequence output in the current denoising time step.

[0082] In one embodiment, generating a target video based on a noise frame representation sequence output at the last denoising time step may include: mapping the noise frame representation sequence output at the last denoising time step from a latent space back to an original space through a pre-trained decoder to obtain the target video. The original space may be a space where a source video is located, such as an image space.

[0083] In one embodiment, in each denoising time step, when predicting noise through a noise prediction network, a noise frame representation sequence is input into the noise prediction network, and prompt information is introduced through the cross-attention mechanism of the noise prediction network, and multiple pixel movement paths are introduced through the self-attention mechanism of the noise prediction network. Noise prediction is performed through the noise prediction network to obtain the noise predicted at the current time step.

[0084] In the first denoising time step, the input noise frame representation sequence is the noise frame representation sequence obtained in step 108, and in each denoising time step starting from the second denoising time step, the input noise frame representation sequence is the noise frame representation sequence output in the previous denoising time step. The noise prediction network can be a network obtained by a UNet network based on a diffusion model. Specifically, the two-dimensional convolution of the UNet network of the diffusion model can be changed to a three-dimensional convolution, so that the UNet network can process videos.

[0085] In one embodiment, the noise prediction network may include an input module, M downsampling modules connected in series, N upsampling modules connected in series, and an output module; wherein the input module is connected to the first of the M downsampling modules connected in series, the last of the N upsampling modules connected in series is connected to the output module, the mth downsampling module among the M downsampling modules is connected to the (N-m+1)th upsampling module among the N upsampling modules, and M and N are equal and are both positive integers greater than 1. In this embodiment, according to the noise frame representation sequence, prompt information, and multiple pixel movement paths, the step of iteratively predicting noise for multiple denoising time steps through the noise prediction network includes:

[0086] In the iterative prediction noise of each denoising time step, the noise frame representation sequence outputted in the previous denoising time step is inputted into the input module for feature extraction, and a feature sequence consisting of the extracted features corresponding to each noise frame representation in the noise frame representation sequence is outputted.

[0087] Through the first one of the M downsampling modules, features are extracted based on the feature sequence output by the input module, the prompt information and the multiple pixel movement paths, and the extracted feature sequence is output.

[0088] Starting from the second one of the M downsampling modules, each module in the M-1 downsampling modules and each module in the N upsampling modules connected in series are determined as the current module in turn, and the features output by the previous module connected to the current module are determined as the input features. Through the current module, features are extracted based on the input features, prompt information and multiple pixel movement paths for feature processing, and the processed features are output; the features output by the Nth one of the N upsampling modules are input to the output module for processing to obtain the noise predicted at the denoising time step.

[0089] Among them, the input module may include multiple convolution layers for performing convolution processing on the input data. The M downsampling modules connected in series refer to the M downsampling modules connected in sequence from front to back according to the data output direction. The downsampling module can be used to extract features from the data input to the downsampling module. The N upsampling modules connected in series refer to the N upsampling modules connected in sequence from front to back according to the data output direction. The upsampling module can be used to perform feature fusion on the data input to the upsampling module. The output module may include multiple convolution layers for performing convolution processing on the data input to the output module.

[0090] In an exemplary embodiment, the noise prediction network includes an input module, M down-sampling modules connected in series, N up-sampling modules connected in series, and an output module; M and N are equal and are both positive integers greater than 1; Figure 2As shown, the above-mentioned step of iteratively predicting noise for multiple denoising time steps through a noise prediction network according to the noise frame representation sequence, prompt information and multiple pixel movement paths may include steps 202 to 212:

[0091] Step 202, in the iterative prediction noise of each denoising time step, the noise frame representation sequence outputted in the previous denoising time step is inputted into the input module for feature extraction, and a feature sequence consisting of the extracted features corresponding to each noise frame representation in the noise frame representation sequence is outputted.

[0092] Among them, when the denoising time step is the first denoising time step, the noise frame representation sequence outputted in the previous denoising time step refers to the noise frame representation sequence obtained in step 108. It can be understood that in the noise prediction network, the feature sequence outputted by each module is composed of features corresponding to each noise frame representation in the noise frame representation sequence, and it can also be understood that the feature sequence is composed of features corresponding to each video frame in the source frame sequence.

[0093] The feature sequence can be recorded as ,in, It can be the feature corresponding to the first video frame, and so on. It may be the feature corresponding to the Kth video frame, K may be the number of video frames in the source frame sequence, H may be the height of each feature, W may be the width of each feature, and C may be the number of channels of each feature.

[0094] Step 204 , extracting features based on the feature sequence output by the input module, the prompt information, and the multiple pixel movement paths through the first one of the M downsampling modules, and outputting the extracted feature sequence.

[0095] Exemplarily, the downsampling module may include a convolution unit, an optical flow attention unit, a cross-attention unit and a downsampling unit. The electronic device may perform feature extraction on the feature sequence output by the input module through the convolution unit of the first of the M downsampling modules, and output the extracted convolution feature sequence; perform optical flow self-attention processing based on the convolution feature sequence and multiple pixel movement paths through the optical flow attention unit of the first of the M downsampling modules to obtain an optical flow attention feature sequence; perform cross-attention processing based on the optical flow attention feature sequence and prompt information through the cross-attention unit of the first of the M downsampling modules to obtain a cross-attention feature sequence; perform downsampling processing based on the cross-attention feature sequence through the downsampling unit of the first of the M downsampling modules to obtain the feature sequence output by the first of the M downsampling modules.

[0096] Among them, the convolution unit can be a unit that extracts convolutional features based on residual connection. The convolution unit in the downsampling module can be a unit obtained from ResnetBlock2D (residual block) in CrossAttnDownBlock2D (cross attention downsampling block) in the UNet network based on the diffusion model, and can be a unit that changes the two-dimensional convolution in the ResnetBlock2D into a three-dimensional convolution. The cross attention unit in the downsampling module can be a CrossAttention block in CrossAttnDownBlock2D (cross attention downsampling block) in the UNet network based on the diffusion model. The downsampling unit can be a unit obtained from Downsample2D (downsampling block) in CrossAttnDownBlock2D (cross attention downsampling block) in the UNet network based on the diffusion model, and can be a unit that changes the two-dimensional convolution in the Downsample2D into a three-dimensional convolution. The optical flow attention unit can be a self-attention unit based on optical flow.

[0097] Step 206 , starting from the second one of the M downsampling modules, through the mth one of the M downsampling modules, extract features based on the feature sequence output by the previous downsampling module, the prompt information and the multiple pixel movement paths, and output the extracted feature sequence.

[0098] Wherein, m is any positive integer from 2 to M. It can be understood that the implementation of step 206 is similar to the implementation of step 204. Specifically, the "first of the M downsampling modules" in various implementations (examples) of step 204 can be adaptively replaced with "the mth of the M downsampling modules", and the "feature sequence output by the input module" in various implementations of step 204 can be adaptively replaced with "the feature sequence output by the previous downsampling module", so as to form various implementations of step 206.

[0099] Step 208 , the first one of the N up-sampling modules performs feature fusion based on the feature sequence output by the Mth one of the M down-sampling modules, the prompt information, and the multiple pixel movement paths, and outputs the fused feature sequence.

[0100] Exemplarily, the upsampling module may include a convolution unit, an optical flow attention unit, a cross attention unit, and an upsampling unit. The electronic device may perform downsampling processing, feature extraction, and upsampling processing on the feature sequence output by the Mth of the M downsampling modules in sequence to obtain an intermediate feature sequence, perform feature fusion on the intermediate feature sequence and the feature sequence output by the Mth of the M downsampling modules to obtain a fused feature sequence, and may perform convolution processing on the fused feature sequence through the first convolution unit of the N upsampling modules, and output the processed convolution feature sequence; perform optical flow self-attention processing based on the convolution feature sequence and multiple pixel movement paths through the first optical flow attention unit of the N upsampling modules to obtain an optical flow attention feature sequence; perform cross-attention processing based on the optical flow attention feature sequence and prompt information through the first cross-attention unit of the N upsampling modules to obtain a cross-attention feature sequence; perform upsampling processing based on the cross-attention feature sequence through the upsampling unit in the first of the N upsampling modules to obtain the feature sequence output by the first of the N upsampling modules.

[0101] Among them, the convolution unit in the upsampling module can be a unit obtained from ResnetBlock2D (residual block) in CrossAttnUpBlock2D (cross attention upsampling block) in the UNet network based on the diffusion model, and specifically, the two-dimensional convolution in the ResnetBlock2D can be changed to a three-dimensional convolution. The cross-attention unit in the upsampling module can be a CrossAttention block in CrossAttnUpBlock2D (cross attention downsampling block) in the UNet network of the diffusion model. The upsampling unit can be a unit obtained from Upsample2D (upsample block) in CrossAttnUpBlock2D (cross attention upsampling block) in the UNet network based on the diffusion model, and specifically, the two-dimensional convolution in the Upsample2D can be changed to a three-dimensional convolution.

[0102] Step 210, starting from the second of the N upsampling modules, through the nth of the N upsampling modules, based on the feature sequence output by the previous upsampling module, the feature sequence output by the (M-n+1)th of the M downsampling modules, the prompt information and multiple pixel movement paths, feature fusion is performed, and the fused feature sequence is output.

[0103] Wherein, n is any positive integer from 2 to N. For M-n+1, ​​for example, M and N can both be 10, n can be 3, and then M-n+1 can be 8.

[0104] Exemplarily, starting from the second of the N upsampling modules, the feature sequence output by the previous upsampling module is fused with the feature sequence output by the (M-n+1)th of the M downsampling modules through the nth of the N upsampling modules to obtain a fused feature sequence, and the fused feature sequence is convolved by the nth of the N upsampling modules, and the processed convolution feature sequence is output; the optical flow self-attention processing is performed by the nth of the N upsampling modules based on the convolution feature sequence and multiple pixel movement paths to obtain an optical flow attention feature sequence; the cross-attention processing is performed by the nth of the N upsampling modules based on the optical flow attention feature sequence and prompt information to obtain a cross-attention feature sequence; the upsampling unit in the nth of the N upsampling modules performs upsampling processing based on the cross-attention feature sequence to obtain a feature sequence output by the nth of the N upsampling modules.

[0105] Step 212, input the feature sequence of the Nth output of the N upsampling modules to the output module for processing to obtain the noise predicted at the denoising time step.

[0106] Exemplarily, the feature sequence of the Nth output of the N upsampling modules is input to the output module for convolution processing to obtain the noise predicted at the denoising time step.

[0107] In this embodiment, multiple feature extractions are performed through M downsampling modules, and in the feature extraction process, feature extraction is performed based on the feature sequence output by the previous module, based on prompt information and multiple pixel movement paths, so that more accurate features can be extracted. Moreover, when multiple feature fusions are performed through N downsampling modules, the features output by the (M-n+1)th downsampling module are fused, and multi-level fusion can be performed to improve the accuracy of the noise of the final output, thereby subsequently improving the denoising accuracy.

[0108] In one embodiment, the convolution feature sequence output by the first convolution unit of the M downsampling modules, in which each convolution feature may include patches at different positions; the step of performing optical flow self-attention processing based on the convolution feature sequence and multiple pixel movement paths by the first optical flow attention unit of the M downsampling modules to obtain the optical flow attention feature sequence may include:

[0109] The electronic device can determine the patch movement path that matches the pixel movement path in the convolution feature sequence according to each pixel movement path in multiple pixel movement paths; determine a query vector for each patch block contained in each convolution feature in the convolution feature sequence based on preconfigured query parameters and the targeted patch block; determine a key vector based on preconfigured key parameters, some patch blocks on the patch movement path where the targeted patch block is located, except for the targeted patch block, and some patch blocks in the neighborhood of the convolution feature where the patch blocks are located; determine a value vector based on preconfigured value parameters, some patch blocks and patch blocks in the neighborhood; perform attention calculation based on the query vector, the key vector and the value vector to determine the attention calculation result of the targeted patch block; splice the attention calculation results of each patch block contained in the convolution feature to obtain an optical flow attention feature corresponding to the convolution feature, so as to obtain an optical flow attention feature sequence composed of each optical flow attention feature.

[0110] The query parameter can be , the key parameter can be the attention mechanism , the value parameter can be the attention mechanism The query parameter, key parameter, and value parameter can also be referred to as the weight matrix of Q (Query), K (Key), and V (Value).

[0111] A patch block is a local area feature in a feature. For example, when the width and height of a feature are 32*32, the feature can include 1024 patches when the patch size is set to 1, and the feature can include 256 patches when the patch size is set to 2. It can be understood that each patch block has a position in the feature, which is similar to the position of a pixel. For example, when the width and height of a feature are 32*32 and the width and height of a video frame are 32*32, the position of the patch in the feature can correspond one-to-one to the position of the pixel in the video frame.

[0112] When the width and height of the feature where the patch block is located are the same as the width and height of the data where the pixel movement path is located (which can be a video frame or a video frame whose size has been reduced as mentioned in step 106), the patch movement path that matches the pixel movement path can be the same path as the pixel movement path. When the width and height of the feature where the patch block is located are smaller than the width and height of the data where the pixel movement path is located, the patch movement path that matches the pixel movement path can be the pixel movement path obtained by downsampling the pixel movement path to the size of the feature where the patch block is located.

[0113] The above-mentioned partial patch blocks in the convolution feature sequence can be the convolution features corresponding to the video frames arranged at the preset positions in the source frame sequence, the patch moving path where the targeted patch blocks are located, and the patch blocks other than the targeted patch blocks. The preset position is a pre-set position, for example, it can be an even position, or an odd position. Even numbers are 2, 4, 6, 8, etc., and odd numbers are 1, 3, 5, 7, etc. Specifically, the preset position can be set to an even position in the odd-numbered denoising time step, and the preset position can be set to an odd position in the even-numbered denoising time step, or the settings can be swapped. The preset position can also be the next even position of the position of the video frame corresponding to the convolution feature where the targeted patch block is located, or the next odd position of the position of the video frame corresponding to the convolution feature where the targeted patch block is located. In actual scenarios, it is observed that there is redundancy between adjacent video frames, so even frames or odd frames can be taken for reference.

[0114] The patch blocks in the neighborhood mentioned above in the convolution feature are patches in the neighborhood of each patch block in the partial patch block in the convolution feature. The neighborhood refers to the neighborhood of the patch block in the feature. The neighborhood can be a 4-neighborhood, an 8-neighborhood, etc. For example, if the neighborhood is an 8-neighborhood, the patch blocks in the neighborhood can be patches at 8 positions around a patch block in the partial patch block.

[0115] In an exemplary embodiment, the downsampling module includes a convolution unit, a spatiotemporal attention unit, an optical flow attention unit, a cross-attention unit and a downsampling unit. The above step 204 may also include: extracting features from the feature sequence output by the input module through the first convolution unit of the M downsampling modules, and outputting the extracted convolution feature sequence; performing spatiotemporal self-attention processing based on the convolution feature sequence through the first spatiotemporal attention unit of the M downsampling modules to obtain a spatiotemporal attention feature sequence; performing optical flow self-attention processing based on the spatiotemporal attention feature sequence and multiple pixel movement paths through the first optical flow attention unit of the M downsampling modules to obtain an optical flow attention feature sequence; performing cross-attention processing based on the optical flow attention feature sequence and prompt information through the first cross-attention unit of the M downsampling modules to obtain a cross-attention feature sequence; performing downsampling processing based on the cross-attention feature sequence through the downsampling unit in the first of the M downsampling modules to obtain the feature sequence output by the first of the M downsampling modules.

[0116] Among them, the spatiotemporal attention unit is a self-attention unit based on temporal and spatial attention.

[0117] In this embodiment, before being processed by the optical flow attention unit, the convolution feature sequence output by the convolution unit is subjected to spatiotemporal self-attention processing by the spatiotemporal attention unit. The obtained spatiotemporal attention feature sequence can pay attention to the temporal information and spatial information of the source video, that is, pay attention to the timing information of each video frame and the picture information of each video frame, which can increase the amount of feature information. Based on the processing by the optical flow attention unit, further attention is paid to the key feature information, that is, attention is paid to the change of pixel position, which can improve the accuracy of the noise of the final output, thereby improving the denoising accuracy subsequently.

[0118] In one embodiment, in the convolution feature sequence output by the first convolution unit in the M downsampling modules, each convolution feature may include patches at different positions, and the step of performing spatiotemporal self-attention processing based on the convolution feature sequence by the first spatiotemporal attention unit in the M downsampling modules to obtain the spatiotemporal attention feature sequence may include:

[0119] For each patch block contained in each convolution feature in the convolution feature sequence, a query vector is determined based on the preconfigured query parameters and the targeted patch block; a key vector is determined based on the preconfigured key parameters and the patch blocks contained in the convolution feature sequence, except the targeted patch block; a value vector is determined based on the preconfigured value parameters and other patch blocks; attention calculation is performed according to the query vector, the key vector and the value vector to determine the attention calculation result of the targeted patch block; the attention calculation results of each patch block contained in the convolution feature are spliced ​​to obtain the spatiotemporal attention feature corresponding to the convolution feature, so as to obtain a spatiotemporal attention feature sequence composed of each spatiotemporal attention feature.

[0120] In an exemplary embodiment, the spatiotemporal attention feature sequence includes a spatiotemporal attention feature corresponding to each noise frame in the noise frame sequence; each spatiotemporal attention feature includes patches at different positions; the step of performing optical flow self-attention processing based on the spatiotemporal attention feature sequence and multiple pixel movement paths by the first optical flow attention unit in the M downsampling modules to obtain the optical flow attention feature sequence may include:

[0121] According to each pixel movement path in the multiple pixel movement paths, determine the patch movement path in the spatiotemporal attention feature sequence that matches the pixel movement path; for each patch block contained in each spatiotemporal attention feature in the spatiotemporal attention feature sequence, determine the query vector based on the preconfigured query parameters and the patch block; based on the preconfigured key parameters, the key vector is determined for some patch blocks other than the patch block on the patch movement path where the patch block is located, and the patch blocks in the neighborhood of some patch blocks in their respective spatiotemporal attention features; based on the preconfigured value parameters, the value vector is determined for some patch blocks and the patch blocks in the neighborhood; attention calculation is performed according to the query vector, the key vector and the value vector to determine the attention calculation result of the patch block; the attention calculation results of each patch block contained in the spatiotemporal attention feature are spliced ​​to obtain the optical flow attention feature corresponding to the spatiotemporal attention feature, so as to obtain the optical flow attention feature sequence composed of each optical flow attention feature.

[0122] Among them, the above-mentioned partial patch blocks in the spatiotemporal attention feature sequence can be the patch blocks in the spatiotemporal attention feature corresponding to the video frame arranged in the preset position in the source frame sequence, on the patch moving path where the targeted patch block is located, and other than the targeted patch block. The preset position is a pre-set position, for example, it can be an even position, or an odd position. Even numbers are 2, 4, 6, 8, etc., and odd numbers are 1, 3, 5, 7, etc. Specifically, the preset position can be set to an even position in the odd-numbered denoising time step, and the preset position can be set to an odd position in the even-numbered denoising time step, or the settings can be swapped. The preset position can also be the next even position of the position of the video frame corresponding to the spatiotemporal attention feature where the targeted patch block is located, or the next odd position of the position of the video frame corresponding to the spatiotemporal attention feature where the targeted patch block is located. The above-mentioned patch blocks in the neighborhood in the spatiotemporal attention feature are patch blocks in the neighborhood of each patch block in the convolution feature where the patch block is located in the partial patch blocks.

[0123] In this embodiment, the attention calculation can be called the attention calculation based on optical flow. The softmax function is used in the attention calculation to obtain the attention weight. The formula of the softmax function is as shown in formula (3):

[0124] Formula (3)

[0125] in, . can represent the query vector, The targeted patch can be characterized. Can represent query parameters, Characterizable and The product of . can represent the key vector, It can represent the preset position in the source frame sequence The patch moving path of the targeted patch block in the spatiotemporal attention feature corresponding to the video frame, the patch blocks other than the targeted patch block (called partial patch blocks), and the patch blocks in the neighborhood of the partial patch blocks in their respective spatiotemporal attention features. Can be called key parameters. Characterizable and The product of . A vector of characterizable values. Can be called a value parameter. Characterizable and The product of . The dimension that can represent the key vector. It can represent the time step. It can be the noise-added time step during the noise-adding process and the denoising time step during the denoising process.

[0126] In this embodiment, since the key vector and the value vector are determined based on some patch blocks other than the patch block on the patch moving path where the patch block is located, as well as patch blocks in the neighborhood of some patch blocks in their respective spatiotemporal attention features, only some patch blocks are taken on the patch moving path, so that a source video with more video frames can be input for processing during video generation, and the patch blocks in the neighborhood are also referenced, so that more local features are referenced on one feature, which brings more detail clues to video generation and makes the generated video more realistic in detail performance.

[0127] In an exemplary embodiment, step 108 may include: processing each video frame in the source frame sequence into a source frame representation, and obtaining a source frame representation sequence consisting of source frame representations corresponding to each video frame in the source frame sequence; iteratively predicting noise for multiple noise-adding time steps through a noise prediction network according to the source frame representation sequence and multiple pixel movement paths; after predicting the noise of each noise-adding time step, based on the noise predicted at the current noise-adding time step, adding noise to the source frame representation sequence output at the previous noise-adding time step, and obtaining a source frame representation sequence output at the current noise-adding time step; determining each source frame representation in the source frame representation sequence output at the last noise-adding time step as a noise frame representation, and obtaining a noise frame representation sequence consisting of noise frame representations corresponding to each video frame in the source frame sequence.

[0128] Among them, the noise prediction network used in the noise adding process of this embodiment can be the same as the noise prediction network used in the denoising process. In the noise adding process, the way of predicting noise through the noise prediction network can be similar to the way of predicting noise through the noise prediction network in the denoising process; the difference is that in the denoising process, the noise prediction network can predict noise based on the noise frame representation sequence, prompt information and multiple pixel movement paths, while in the noise adding process, the noise prediction network can predict noise based on the source frame representation sequence and multiple pixel movement paths. It can be understood that the noise frame representation sequence in the denoising process can be adaptively replaced with the source frame representation sequence, and the processing of the prompt information in the denoising process can be eliminated, so as to obtain the way of predicting noise through the noise prediction network.

[0129] In this embodiment, according to the source frame representation sequence and multiple pixel movement paths, the noise is iteratively predicted for multiple noise-adding time steps through a noise prediction network. After the noise of each noise-adding time step is predicted, the source frame representation sequence output at the previous noise-adding time step is noised based on the noise predicted at the current noise-adding time step to obtain the source frame representation sequence output at the current noise-adding time step. In this way, by predicting the noise through the noise prediction network and utilizing the advantages of the noise prediction network, the timing information of each video frame in the input source video can be better preserved, thereby optimizing the timing consistency of the subsequently generated target video.

[0130] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the indications of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.

[0131] Based on the same inventive concept, the embodiment of the present application also provides a video generation device for implementing the video generation method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more video generation device embodiments provided below can refer to the limitations on the video generation method above, and will not be repeated here.

[0132] In an exemplary embodiment, Figure 3As shown, a video generation device 300 is provided, comprising: an acquisition module 310, an optical flow prediction module 320, a path determination module 330 and a video generation module 340, wherein:

[0133] The acquisition module 310 is used to acquire a source frame sequence of a source video and acquire prompt information for the source video; the source frame sequence includes a plurality of continuous video frames in the source video.

[0134] The optical flow prediction module 320 is used to predict the optical flow field of each video frame pair consisting of two adjacent video frames in the source frame sequence.

[0135] The path determination module 330 is used to determine multiple pixel movement paths of the source frame sequence based on the optical flow field; the pixel movement path represents the position change of the same pixel in at least two consecutive video frames in the source frame sequence.

[0136] The video generation module 340 is used to add noise to each video frame in the source frame sequence to obtain a noise frame representation, and generate a noise frame representation sequence consisting of the noise frame representation corresponding to each video frame in the source frame sequence; iteratively predict the noise according to the noise frame representation sequence, prompt information and multiple pixel movement paths, and iteratively denoise the noise frame representation sequence based on the predicted noise to generate a target video.

[0137] In one embodiment, the video generation module 340 is also used to iteratively predict the noise for multiple denoising time steps through a noise prediction network based on the noise frame representation sequence, prompt information and multiple pixel movement paths; after predicting the noise of each denoising time step, based on the noise predicted in the current denoising time step, denoise the noise frame representation sequence output in the previous denoising time step to obtain the noise frame representation sequence output in the current denoising time step; and generate the target video based on the noise frame representation sequence output in the last denoising time step.

[0138] In one embodiment, the noise prediction network includes an input module, M down-sampling modules connected in series, N up-sampling modules connected in series, and an output module; M is equal to N and both are positive integers greater than 1; the video generation module 340 is also used to input the noise frame representation sequence outputted in the previous denoising time step into the input module for feature extraction in the iterative prediction noise of each denoising time step, and output a feature sequence composed of the features corresponding to each noise frame representation in the noise frame representation sequence; extract features based on the feature sequence outputted by the input module, prompt information, and multiple pixel movement paths through the first of the M down-sampling modules, and output the extracted feature sequence; starting from the second of the M down-sampling modules, through the mth of the M down-sampling modules, based on the previous down-sampling The feature sequence, prompt information and multiple pixel movement paths output by the module are used to extract features, and the extracted feature sequence is output; through the first of the N upsampling modules, feature fusion is performed based on the feature sequence, prompt information and multiple pixel movement paths output by the Mth out of the M downsampling modules, and the fused feature sequence is output; starting from the second of the N upsampling modules, through the nth of the N upsampling modules, feature fusion is performed based on the feature sequence output by the previous upsampling module, the feature sequence, prompt information and multiple pixel movement paths output by the (M-n+1)th out of the M downsampling modules, and the fused feature sequence is output; the feature sequence output by the Nth out of the N upsampling modules is input to the output module for processing to obtain the noise predicted at the denoising time step.

[0139] In one embodiment, the downsampling module includes a convolution unit, a spatiotemporal attention unit, an optical flow attention unit, a cross attention unit and a downsampling unit. The video generation module 340 is also used to extract features from the feature sequence output by the input module through the first convolution unit of the M downsampling modules, and output the extracted convolution feature sequence; perform spatiotemporal self-attention processing based on the convolution feature sequence through the first spatiotemporal attention unit of the M downsampling modules to obtain a spatiotemporal attention feature sequence; perform optical flow self-attention processing based on the spatiotemporal attention feature sequence and multiple pixel movement paths through the first optical flow attention unit of the M downsampling modules to obtain an optical flow attention feature sequence; perform cross-attention processing based on the optical flow attention feature sequence and prompt information through the first cross-attention unit of the M downsampling modules to obtain a cross-attention feature sequence; perform downsampling processing based on the cross-attention feature sequence through the downsampling unit of the first downsampling module of the M downsampling modules to obtain the feature sequence output by the first downsampling module of the M downsampling modules.

[0140] In one embodiment, the spatiotemporal attention feature sequence includes a spatiotemporal attention feature corresponding to each noise frame in the noise frame sequence; each spatiotemporal attention feature includes patch blocks at different positions; the video generation module 340 is also used to determine the patch movement path in the spatiotemporal attention feature sequence that matches the pixel movement path according to each pixel movement path in the multiple pixel movement paths; for each patch block contained in each spatiotemporal attention feature in the spatiotemporal attention feature sequence, based on preconfigured query parameters and patch blocks, a query vector is determined; based on preconfigured key parameters, the spatiotemporal attention feature sequence contains some patch blocks other than the patch block on the patch movement path where the patch block is located, and the patch blocks in the neighborhood of the some patch blocks in their respective spatiotemporal attention features, a key vector is determined; based on preconfigured value parameters, some patch blocks and the patch blocks in the neighborhood, a value vector is determined; attention calculation is performed according to the query vector, the key vector and the value vector to determine the attention calculation result of the patch block; the attention calculation results of each patch block contained in the spatiotemporal attention feature are spliced ​​to obtain an optical flow attention feature corresponding to the spatiotemporal attention feature, so as to obtain an optical flow attention feature sequence composed of each optical flow attention feature.

[0141] In one embodiment, the video generation module 340 is also used to process each video frame in the source frame sequence into a source frame representation, and obtain a source frame representation sequence consisting of the source frame representation corresponding to each video frame in the source frame sequence; according to the source frame representation sequence and multiple pixel movement paths, perform iterative noise prediction for multiple noise-adding time steps through a noise prediction network; after predicting the noise of each noise-adding time step, add noise to the source frame representation sequence output at the previous noise-adding time step based on the noise predicted at the current noise-adding time step, and obtain the source frame representation sequence output at the current noise-adding time step; determine each source frame representation in the source frame representation sequence output at the last noise-adding time step as a noise frame representation, and obtain a noise frame representation sequence consisting of the noise frame representation corresponding to each video frame in the source frame sequence.

[0142] Each module in the above video generation device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in an electronic device in the form of hardware, or can be stored in a memory in an electronic device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0143] In an exemplary embodiment, an electronic device is provided. The electronic device may be a server, and its internal structure diagram may be as shown in FIG. Figure 4As shown. The electronic device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the electronic device is used to store data to be stored when executing the above-mentioned video generation method. The input / output interface of the electronic device is used to exchange information between the processor and an external device. The communication interface of the electronic device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a video generation method is implemented.

[0144] Those skilled in the art will understand that Figure 4 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0145] In one embodiment, an electronic device is further provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned method embodiments when executing one or more programs.

[0146] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned method embodiments are implemented.

[0147] In one embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the above-mentioned method embodiments are implemented.

[0148] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0149] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0150] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0151] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A video generation method, characterized in that: The method comprises: Acquire a source frame sequence of a source video, and acquire prompt information for the source video; the source frame sequence includes a plurality of continuous video frames in the source video; For each video frame pair consisting of two adjacent video frames in the source frame sequence, predicting the optical flow field of the video frame pair; Determine a plurality of pixel movement paths of the source frame sequence based on the optical flow field; the pixel movement paths represent position changes of the same pixel in at least two consecutive video frames in the source frame sequence; Noising each video frame in the source frame sequence to obtain a noise frame representation, and generating a noise frame representation sequence consisting of the noise frame representation corresponding to each video frame in the source frame sequence; According to the noise frame representation sequence, the prompt information and the multiple pixel movement paths, noise is iteratively predicted, and the noise frame representation sequence is iteratively denoised based on the predicted noise to generate a target video.

2. The method according to claim 1, characterized in that The method of iteratively predicting noise according to the noise frame representation sequence, the prompt information and the plurality of pixel movement paths, iteratively denoising the noise frame representation sequence based on the predicted noise, and generating a target video includes: Iteratively predicting noise at multiple denoising time steps through a noise prediction network according to the noise frame representation sequence, the prompt information, and the multiple pixel movement paths; After the noise of each denoising time step is predicted, the noise frame representation sequence outputted at the previous denoising time step is denoised based on the noise predicted at the current denoising time step, and the noise frame representation sequence outputted at the current denoising time step is obtained; Generate the target video based on the sequence of noisy frame representations output at the last denoising time step.

3. The method according to claim 2, characterized in that The noise prediction network includes an input module, M downsampling modules connected in series, N upsampling modules connected in series, and an output module; M and N are equal and are both positive integers greater than 1; The iterative prediction of noise at multiple denoising time steps is performed through a noise prediction network according to the noise frame representation sequence, the prompt information and the multiple pixel movement paths, including: In the iterative prediction noise of each denoising time step, the noise frame representation sequence outputted in the previous denoising time step is inputted into the input module for feature extraction, and a feature sequence consisting of the extracted features corresponding to each noise frame representation in the noise frame representation sequence is outputted; Extracting features based on the feature sequence output by the input module, the prompt information and the plurality of pixel movement paths through a first one of the M downsampling modules, and outputting the extracted feature sequence; Starting from the second one of the M down-sampling modules, extracting features based on the feature sequence output by the previous down-sampling module, the prompt information and the plurality of pixel movement paths through the mth one of the M down-sampling modules, and outputting the extracted feature sequence; Performing feature fusion based on the feature sequence output by the Mth one of the M down-sampling modules, the prompt information, and the plurality of pixel movement paths through the first one of the N up-sampling modules, and outputting the fused feature sequence; Starting from the second one of the N up-sampling modules, through the nth one of the N up-sampling modules, based on the feature sequence output by the previous up-sampling module, the feature sequence output by the (M-n+1)th one of the M down-sampling modules, the prompt information and the multiple pixel movement paths, feature fusion is performed, and a fused feature sequence is output; The feature sequence of the Nth output of the N upsampling modules is input into the output module for processing to obtain the noise predicted at the denoising time step.

4. The method according to claim 3, characterized in that The downsampling module includes a convolution unit, a spatiotemporal attention unit, an optical flow attention unit, a cross attention unit and a downsampling unit. The first of the M downsampling modules is used to extract features based on the feature sequence output by the input module, the prompt information and the plurality of pixel movement paths, and outputting the extracted feature sequence includes: Extracting features from the feature sequence output by the input module through the first convolution unit of the M downsampling modules, and outputting the extracted convolution feature sequence; Performing spatiotemporal self-attention processing based on the convolution feature sequence through the first spatiotemporal attention unit in the M downsampling modules to obtain a spatiotemporal attention feature sequence; Performing optical flow self-attention processing based on the spatiotemporal attention feature sequence and the multiple pixel movement paths through the first optical flow attention unit in the M downsampling modules to obtain an optical flow attention feature sequence; Performing cross-attention processing based on the optical flow attention feature sequence and the prompt information through the first cross-attention unit in the M downsampling modules to obtain a cross-attention feature sequence; Downsampling processing is performed based on the cross-attention feature sequence through a downsampling unit in the first one of the M downsampling modules to obtain a feature sequence output by the first one of the M downsampling modules.

5. The method according to claim 4, characterized in that The spatiotemporal attention feature sequence includes a spatiotemporal attention feature corresponding to each noise frame in the noise frame sequence; each spatiotemporal attention feature includes a patch block at a different position; The first optical flow attention unit in the M downsampling modules performs optical flow self-attention processing based on the spatiotemporal attention feature sequence and the multiple pixel movement paths to obtain an optical flow attention feature sequence, including: According to each pixel movement path in the multiple pixel movement paths, determining a patch movement path in the spatiotemporal attention feature sequence that matches the pixel movement path; For each patch block included in each spatiotemporal attention feature in the spatiotemporal attention feature sequence, determine a query vector based on preconfigured query parameters and the patch block; Determine a key vector based on preconfigured key parameters, the partial patch blocks on the patch moving path where the patch block is located except the patch block contained in the spatiotemporal attention feature sequence, and the patch blocks in the neighborhood of the partial patch blocks in the spatiotemporal attention features where the patch block is located; Determine a value vector based on preconfigured value parameters, the partial patch blocks and the patch blocks in the neighborhood; Performing attention calculation according to the query vector, the key vector, and the value vector to determine an attention calculation result of the patch block; The attention calculation results of each patch block contained in the spatiotemporal attention feature are spliced ​​to obtain the optical flow attention feature corresponding to the spatiotemporal attention feature, so as to obtain an optical flow attention feature sequence composed of the optical flow attention features.

6. The method according to any one of claims 1 to 5, characterized in that: The step of adding noise to each video frame in the source frame sequence to obtain a noise frame representation, and generating a noise frame representation sequence consisting of the noise frame representation corresponding to each video frame in the source frame sequence, comprises: Processing each video frame in the source frame sequence into a source frame representation, obtaining a source frame representation sequence consisting of source frame representations corresponding to each video frame in the source frame sequence; According to the source frame representation sequence and the plurality of pixel movement paths, performing iterative noise prediction for a plurality of noise adding time steps through a noise prediction network; After predicting the noise of each noise adding time step, based on the noise predicted in the current noise adding time step, the source frame representation sequence output in the previous noise adding time step is noised to obtain the source frame representation sequence output in the current noise adding time step; Each source frame representation in the source frame representation sequence output at the last noise addition time step is determined as a noise frame representation, and a noise frame representation sequence consisting of the noise frame representation corresponding to each video frame in the source frame sequence is obtained.

7. A video generating device, characterized in that: The device comprises: An acquisition module, used to acquire a source frame sequence of a source video and acquire prompt information for the source video; the source frame sequence includes a plurality of continuous video frames in the source video; An optical flow prediction module, configured to predict an optical flow field of each video frame pair consisting of two adjacent video frames in the source frame sequence; A path determination module, configured to determine a plurality of pixel movement paths of the source frame sequence based on the optical flow field; the pixel movement paths represent position changes of the same pixel in at least two consecutive video frames in the source frame sequence; The video generation module is used to add noise to each video frame in the source frame sequence to obtain a noise frame representation, and generate a noise frame representation sequence consisting of the noise frame representation corresponding to each video frame in the source frame sequence; iteratively predict noise according to the noise frame representation sequence, the prompt information and the multiple pixel movement paths, and iteratively denoise the noise frame representation sequence based on the predicted noise to generate a target video.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes one or more programs, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images