A controllable generative video frame insertion method based on diffusion model

Abstract

The present invention discloses a controllable generative video interpolation method based on a diffusion model, including: based on a graph-generated video diffusion model, introducing a tail frame control condition to realize video interpolation; introducing a trajectory control scheme based on user dragging, allowing the user to realize controllable interpolation through simple interaction; in the case where the user does not provide a trajectory, obtaining the matching information of key points between the first and last frames through a feature point matching algorithm, and using this information to obtain a temporally consistent interpolation result; using the similarity between the features in the model to do trajectory update to update the coordinates of the points; and ensuring the accuracy of the updated point coordinates by checking the consistency of the updated points obtained by two nearest neighbor algorithms. The method of the present invention improves the accuracy and controllability of video interpolation, can realize user interactive video interpolation generation, and provides more comprehensive performance guidance.

Description

Technical Field

[0001] The invention belongs to the technical field of video model application, and in particular relates to a controllable generative video frame insertion method based on a diffusion model. Background Art

[0002] In the current field of multimedia technology and artificial intelligence, the development of video processing technology has achieved remarkable achievements, especially in video frame interpolation. Video interpolation is an important task in the field of computer vision and video processing. Its purpose is to synthesize an intermediate frame from two consecutive video frames. Most previous methods regard video interpolation as a low-level visual task, assuming that the motion between frames is relatively small. These methods can be roughly divided into flow-based methods and kernel-based methods. Specifically, flow-based methods use estimated optical flow for frame synthesis, while kernel-based methods rely on spatially adaptive kernels to synthesize interpolated pixels; the former may be affected by inaccurate optical flow estimation, while the latter is often limited by the kernel size. In order to get the best of both worlds, some methods combine end-to-end video interpolation methods based on flow and kernel.

[0003] Recently, inspired by the generative capabilities of large-scale pre-trained video diffusion models, some methods have attempted to address the video interpolation problem from a generative perspective. For example, LDM-VFI (Video Frame Interpolation with LatentDiffusion Models) models video interpolation as a conditional generation problem and uses diffusion models for perceptually guided video interpolation; VIDIM (Video Interpolation With Diffusion Models) uses cascaded diffusion models to generate high-fidelity interpolated videos with nonlinear motion. Although these methods have made progress, they still have difficulty producing reliable video interpolation results when dealing with large differences between the start and end frames; in addition, these methods focus on generating a single feasible solution for video interpolation without controllability over the video interpolation results.

[0004] In the field of text-to-video generation, large-scale pre-trained diffusion models have demonstrated the ability to generate high-quality, diverse, and realistic videos. However, these models are limited in terms of precise text control and user interactivity. Traditional video control methods such as VideoComposer and SparseCtrl use structural information such as sketches and depth maps. Although they provide a certain degree of control, the process of obtaining these control signals is complicated and not easy for users to operate, which limits their widespread use in practical applications. In contrast, motion control such as MotionCtrl is a more intuitive control method, such as controlling the motion trajectory of objects and the motion posture of cameras, which can be achieved through simple user input, greatly improving the interactivity and practicality of video generation.

[0005] Nevertheless, effectively integrating motion control into the video interpolation process of diffusion models to generate videos that both conform to the text description and accurately follow the user's control remains an under-solved technical problem. Summary of the invention

[0006] In view of the above, the present invention provides a controllable generative video interpolation method based on a diffusion model. Through innovative motion control strategies and optimized diffusion model architecture, the accuracy and controllability of video interpolation are improved to adapt to various complex motion scenes and meet user customization needs, thereby promoting the further development of video processing technology.

[0007] A controllable generative video frame insertion method based on a diffusion model comprises the following steps:

[0008] (1) Select a pre-trained diffusion model based on image-generated videos;

[0009] (2) Input the first frame and the last frame provided by the user into the diffusion model;

[0010] (3) Introducing user-dragging trajectory control into the diffusion model to generate video interpolation results that meet the user's intentions;

[0011] (4) When the user does not provide a drag trajectory, the trajectory is generated through feature point matching interpolation and automatic point trajectory tracking to improve the quality and consistency of the video interpolation results.

[0012] Furthermore, the diffusion model in step (1) uses an SVD (Stable Video Diffusion) model to generate video interpolation results, which includes a variational auto-encoder (VAE), a CLIP (Contrastive Language-Image Pre-training) image encoder, a 3D U-Net, and a cross-attention mechanism. The variational auto-encoder is used to extract latent space features of video frames, and the CLIP image encoder is used to extract semantic features of video frames. The latent space features are spliced ​​with noisy latent variables and then input into the 3D U-Net. The cross-attention mechanism uses semantic features as key and value inputs and the internal features of the 3D U-Net as query inputs. The output of the cross-attention mechanism is used to update the internal features of the 3D U-Net. The 3D U-Net is denoised through multiple rounds of iterations and then outputs the video interpolation results. The traditional SVD model uses a single frame as input and uses it as the first frame to infer and generate the video interpolation results.

[0013] Furthermore, in step (2), when the input of the diffusion model is the first and last frames, the first frame and the last frame are used to generate their own latent space features and semantic features through the variational autoencoder and the CLIP image encoder, and then the latent space features of the first frame and the last frame are spliced ​​with the noisy latent variables and input into the 3D U-Net, and the semantic features of the first frame and the last frame are spliced ​​as the input of the key and value of the cross attention mechanism.

[0014] Furthermore, in order to facilitate user interaction in step (3), the user is allowed to control the video interpolation result by dragging, that is, the trajectory of the key points in the dragging process is obtained and converted into a Gaussian heat map, and then the Gaussian heat map is input into the encoding module for encoding to obtain the features of the key point trajectory, and then the features are injected into the 3D U-Net of the diffusion model. By introducing trajectory control, video interpolation can be made more controllable, and users can meet their needs through simple interaction; experimental results show that the introduction of trajectory control can further improve the performance of video interpolation.

[0015] Furthermore, before introducing the trajectory control based on user dragging, it is necessary to obtain the trajectory control conditions of the points for training the diffusion model. Specifically: first, some sampling points are randomly initialized around a fixed sparse grid in the first frame, and Co-Tracker is used to obtain the trajectories of these sampling points in the entire video; during the training process, the invisible trajectories in more than half of the video frames are removed, and the points with large trajectory motion changes are sampled with high probability in the remaining trajectories; after sampling the trajectory points, only a small number of them are retained and the coordinates of these points are converted into Gaussian heat maps as the input of the encoding module, which copies the encoder part of the 3D U-Net, and finally the features output by the encoding module are injected into the 3D U-Net of the diffusion model through zero convolution.

[0016] Furthermore, in step (4), the matching information of key points is obtained between the first frame and the last frame provided by the user through the feature point matching algorithm, and any key point pair that matches between the first frame and the last frame is recorded as p 0 and p n , the corresponding trajectory is obtained through interpolation estimation, and then all the estimated trajectories are obtained by traversing other matching key point pairs. This method can avoid the problem of "frame skipping" when the difference between the previous and next frames is large, generate more coherent video interpolation results, and improve the quality and coherence of video interpolation.

[0017] Furthermore, the automated point trajectory tracking in step (4) is to update the point coordinates of the trajectory in the intermediate frame by using the similarity between the features in the diffusion model 3D U-Net for the trajectory generated by the feature point matching interpolation. Specifically, the trajectory generated by the feature point matching interpolation is first converted into a Gaussian heat map and then encoded to obtain the trajectory features, and then the trajectory features are injected into the 3D U-Net of the diffusion model; then the features of the second to last upsampling module in the 3D U-Net are used to obtain the image features of each frame through bilateral linear interpolation; for any trajectory point on the intermediate frame, k , by searching for the point k Distance less than threshold r 1 point set Ω; then use the nearest neighbor algorithm to calculate and obtain the key points of the first frame from the point set Ω p 0 The point closest to the feature p k,0 And the key points of the last frame p n The point closest to the feature p k,n , if point p k,0 and p k,n The distance is less than the thresholdr 2, then the trajectory point k Coordinates updated to points p k,0 and p k,n The midpoint coordinates between the two frames are obtained, and the trajectory points of the intermediate frames are updated once according to the above operation for each round of 3D U-Net denoising. The updated trajectory features are used as the input for the next round of 3D U-Net denoising. This method can improve the accuracy and coherence of video frame insertion by further improving the accuracy of point trajectory updates.

[0018] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor is used to execute the computer program to implement the above-mentioned controllable generative video frame insertion method based on the diffusion model.

[0019] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the controllable generative video frame insertion method based on a diffusion model is implemented.

[0020] The creativity of the controllable generative video frame insertion method of the present invention is mainly reflected in the following key technical points:

[0021] 1. Model structure and introduction of tail frame conditions: Based on the graph-generated video diffusion model, this paper introduces tail frame control conditions to achieve video interpolation. By injecting tail frame conditions into latent space and semantic space, users can generate multiple feasible video interpolation schemes given the first and last frames. This method can effectively utilize the prior knowledge of the pre-trained model and improve the quality and diversity of video interpolation.

[0022] 2. Controllable video interpolation: The present invention introduces a trajectory control scheme based on user dragging, allowing users to achieve controllable interpolation through simple interaction. By randomly initializing sampling points in the first frame and using the video point trajectory tracking algorithm to obtain the trajectory of these points in the entire video, users can control the result of video interpolation. This method allows users to easily control the content and style of video interpolation, improving the controllability and user experience of video interpolation.

[0023] 3. Explicit association modeling based on matching points: When the user does not provide a trajectory, the present invention obtains the matching information of key points between the first and last frames through a feature point matching algorithm, and uses this information to obtain a temporally consistent interpolation result; this method can effectively solve the "frame skipping" problem in video interpolation and improve the quality and coherence of video interpolation.

[0024] 4. Automated point trajectory tracking: The present invention uses the similarity between features in 3D U-Net to update the coordinates of the points and ensure the accuracy of the point trajectory. This method can effectively improve the accuracy and robustness of point trajectory tracking and improve the quality and reliability of video interpolation.

[0025] 5. Trajectory consistency check: The present invention ensures the accuracy of the updated point coordinates by checking the consistency of the updated points obtained by the two nearest neighbor algorithms; this method can effectively improve the accuracy and reliability of point trajectory updates and improve the quality and reliability of video interpolation.

[0026] It can be seen that compared with the prior art, the present invention can better handle the huge difference between the start frame and the end frame and generate more reliable video interpolation results; in addition, the method of the present invention provides better controllability, allowing users to control the video interpolation results through simple input. By using motion control as a more intuitive control method, users can control the video interpolation results through simple input, which greatly improves the interactivity and practicality of video generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 The figure is a schematic diagram of the process of controlling the generation of video interpolation frames using a diffusion model in the present invention.

[0028] Figure 2 This is a schematic diagram of automatic point trajectory tracking using similarities between features in 3D U-Net in the present invention.

[0029] Figure 3 This is the video frame insertion result of the method of the present invention applied in a new viewing angle generation scenario.

[0030] Figure 4 This is the video frame interpolation result of applying the method of the present invention to the comic frame interpolation scenario.

[0031] Figure 5 This is the video frame insertion result of the method of the present invention applied to the time-lapse photography and slow-motion video generation scenarios.

[0032] Figure 6 This is the video frame insertion result of the method of the present invention applied in the image deformation scenario. DETAILED DESCRIPTION

[0033] In order to describe the present invention more specifically, the technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0034] The present invention is a controllable generative video interpolation method based on a diffusion model, which aims to achieve user-interactive video interpolation generation. The method includes the following key technologies: First, the present invention introduces the model structure and how to add the tail frame condition on the basis of the video diffusion model to achieve video interpolation, allowing the user to generate a variety of feasible video interpolation schemes given the first and last frames. Secondly, the present invention introduces a trajectory control scheme based on user dragging, allowing the user to achieve controllable interpolation through simple interaction, further enhancing the controllability of video interpolation. Finally, we found that when the first and last frames are very different, the model is prone to produce frame skipping video results. For this reason, the present invention introduces correlation modeling displayed on key points in the video to obtain temporally coherent interpolation results. Through these technologies, the present invention achieves user-interactive video interpolation generation, improves the stability and controllability of video interpolation, and has broad application prospects. The specific implementation process is as follows:

[0035] (1) Establishing the model structure: SVD is a commonly used image-based video diffusion model that can generate high-quality video results based on the first frame of the input image. SVD follows the paradigm of latent diffusion. Specifically, the video is compressed into a lower-dimensional latent space by VAE encoding and decoding. The encoding and decoding process can be expressed as ; In the reverse process, SVD uses 3D U-Net for denoising. In this process, it adopts two strategies to introduce the first frame condition: first, the latent space features encoded by VAE are transformed into The hidden variables with noise in each frame of video are connected respectively. In addition, the semantic features of the first frame are extracted through the CLIP image encoder and injected into the model through the cross attention mechanism. 3D-Unet is trained by DSM (Denoising Score Matching), and the training objective can be expressed as:

[0036]

[0037] (2) Introduction and fine-tuning of the tail frame condition: Based on the graph-generated video diffusion model, we need to introduce the tail frame control to realize video interpolation. To this end, this paper introduces the tail frame control condition in SVD. In order to retain the prior of pre-trained SVD as much as possible, we inject the tail frame condition in the latent space and semantic space respectively. Specifically, we transform the latent space features of the first frame into Connect with the noisy latent variable of the first frame and encode the latent space features of the last frame after VAE The noisy latent variable of the last frame is connected; in the middle frame, the noisy latent variable is simply connected to the learnable conditional token after broadcast. In addition, the present invention extracts the CLIP image embedding features of the first frame and the last frame respectively and connects them together as the key and value in the cross attention mechanism. The training objective after fine-tuning the model can be expressed as:

[0038]

[0039] (3) Controllable video frame insertion: Given the first and last frames, the conditional probability distribution There can be multiple feasible video interpolation results in the sampled video, especially when there is a large difference between the first and last frames. In order to facilitate user interaction, the present invention considers using a control method such as drag. In order to obtain the trajectory control conditions of the point to train the model, we randomly initialize some sampling points around a fixed sparse grid in the first frame, and use Co-Tracker to obtain the trajectories of these points in the entire video; in training, we remove trajectories that are invisible in more than half of the video frames, and sample large points with higher probability according to trajectory action changes. In addition, considering that the control points input by the user are often sparse, we only retain 1 to 5 control points in training. After sampling the trajectory points, we convert the point coordinates into a Gaussian heat map, which is expressed as , as the input of the control module. This paper adopts a mechanism similar to ControlNet to add trajectory control. Specifically, we copy the encoder of 3D U-Net to encode the trajectory map and inject it into the decoder of U-Net through zero convolution, such as Figure 1 As shown; after the introduction of trajectory control, the training objective of the model can be expressed as:

[0040]

[0041] This loss function takes into account the trajectory control condition to generate a video interpolation result that meets the user's intention. Our trajectory control makes video interpolation more controllable and meets the user's needs through simple interaction. In addition, we also found in the experiment that the introduction of trajectory control can further improve the performance of video interpolation.

[0042] (4) Explicit association modeling based on matching points: When the user does not provide a trajectory, although the method of the present invention can achieve good interpolation effects when the difference between the previous and next frames is moderate or small, we found that when the difference between the previous and next frames is large, "frame skipping" sometimes occurs. That is, the first half of the generated interpolation video is highly correlated with the first frame, and the second half is highly correlated with the last frame. In the middle frame, the video will suddenly change from content related to the first frame to content related to the last frame.

[0043] Given the first and last frames of a given input video, the present invention can obtain the matching information of key points between the first and last frames through a feature point matching algorithm. To indicate that is the number of matching points, is a known point on the trajectory (at initialization, it is the key point matched on the first and last frames). Although different key point matching algorithms are feasible, this embodiment uses the classic SIFT feature point matching because it is simple and we empirically found that it works well.

[0044] Then, we use the known trajectory points Interpolation to obtain the i Trajectory , in this way all estimated trajectories can be obtained .

[0045] (5) Automated point trajectory tracking: Although the initial trajectory can provide time-consistent key point association information, the connection trajectory obtained based on the first and last key frame interpolation may not be accurate; therefore, in each denoising step, the present invention uses the similarity between features in 3D U-Net to perform trajectory updates to update the coordinates of the points.

[0046] Specific as Figure 2 As shown in , we interpolate the features on the 3D U-Net to the image resolution to obtain the graph feature F; here we use the features of the second-to-last upsampling module in the 3D U-Net because it has a higher resolution and better discriminability; Represents the coordinates on the graph feature F p The feature at is obtained by bilateral linear interpolation, because the coordinate point p Not necessarily an integer.

[0047] We search for updated coordinate points near the interpolated intermediate frame coordinates, using Indicates the point p The distance is less than r The set of points with 1, that is , use the nearest neighbor algorithm to obtain k The update point on the frame that is closest to the key point on the first frame:

[0048]

[0049] (6) Trajectory consistency check: In order to further improve the accuracy of point trajectory update, we use the matching point-to-point coordinates of the last frame to update, and then obtain the first k The update point on the frame that is closest to the key point on the last frame:

[0050]

[0051] To ensure the accuracy of the updated point coordinates, we check the consistency of the updated points obtained by the nearest neighbor algorithm twice. r 2, namely When , we use the midpoint of these two points as the updated point coordinates, that is, .

[0052] We then add this point to the known trajectory points , and interpolate to get the updated trajectory , as the input for the next denoising step.

[0053] Based on the above technical solution, the controllable generative video frame insertion method of the present invention has a wide range of application values, including:

[0054] 1. New perspective generation: New perspective generation obtained from sparse perspective input can achieve new perspective generation from static scenes to dynamic scenes; for example, through video interpolation, pictures from different perspectives can be generated as the first and last frames of the video, thereby achieving new perspective generation, such as Figure 3 shown.

[0055] 2. Comic Insertion: By inserting comic pictures, you can get animation, simplify the animation production process, and support the insertion of color comics and line-drawn comics; for example, by inserting comic pictures, you can generate animation videos, and support the insertion of color comics and line-drawn comics, such as Figure 4 shown.

[0056] 3. Video editing: Use video interpolation to implement video editing, including motion editing and video filling, and provide new ideas. For example, through video interpolation, you can implement video motion editing, such as modifying the actions of characters in the video. At the same time, you can also implement video filling, such as deleting certain objects in the video.

[0057] 4. Time-lapse photography: Only a few images of key moments are needed to obtain the entire video through interpolation, showing the slowly changing process; for example, through video interpolation, a time-lapse video can be generated to show the melting of ice, the growth of flowers and plants, and the waxing and waning of the moon. Figure 5 As shown in (a).

[0058] 5. Slow motion video generation: Slow motion video can be generated by video interpolation; for example, converting a normal speed video into a slow motion video to highlight the key frames and actions in the video, such as Figure 5 As shown in (b).

[0059] 6. Interactive video: Interactive video editing can be achieved through video interpolation; for example, users can control the video playback speed, direction and interpolation effect through a mouse or touch screen, achieving a more flexible and interactive video editing experience.

[0060] 7. Image deformation: Through video interpolation, you can generate a gradual change effect from one content to another; for example, a photo of a young person gradually changes to an old state. This technology can be applied to advertising, movie special effects and other fields to create unique visual effects, such as Figure 6 shown.

[0061] The above description of the embodiments is to facilitate the understanding and application of the present invention by those skilled in the art. It is obvious that those skilled in the art can easily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative work. Therefore, the present invention is not limited to the above embodiments. Improvements and modifications made by those skilled in the art to the present invention based on the disclosure of the present invention should be within the protection scope of the present invention.

Claims

1. A controllable generative video frame insertion method based on a diffusion model, comprising the following steps: (1) A pre-trained diffusion model based on image-generated video is selected. The diffusion model uses an SVD model to generate video interpolation results, which includes a variational autoencoder, a CLIP image encoder, a 3D U-Net, and a cross-attention mechanism. The variational autoencoder is used to extract latent space features of video frames. The CLIP image encoder is used to extract semantic features of video frames. The latent space features are spliced ​​with latent variables with noise and then input into the 3D U-Net. The cross-attention mechanism uses semantic features as key and value inputs and the internal features of the 3D U-Net as query inputs. The output of the cross-attention mechanism is used to update the internal features of the 3D U-Net. The 3D U-Net is denoised through multiple rounds of iterations and then outputs the video interpolation results. (2) Input the first frame and the last frame provided by the user into the diffusion model; (3) Introducing user-drag-based trajectory control into the diffusion model to generate video interpolation results that meet the user's intention; (4) When the user does not provide a drag trajectory, the trajectory is generated through feature point matching interpolation and automatic point trajectory tracking.

2. The controllable generative video frame insertion method based on diffusion model according to claim 1, characterized in that: In the step (2), when the input of the diffusion model is the first and last frames, the first frame and the last frame are used to generate their own latent space features and semantic features through a variational autoencoder and a CLIP image encoder, and then the latent space features of the first frame and the last frame are spliced ​​with the noisy latent variables and input into the 3D U-Net, and the semantic features of the first frame and the last frame are spliced ​​as the input of the key and value of the cross attention mechanism.

3. The controllable generative video frame insertion method based on diffusion model according to claim 1, characterized in that: In order to facilitate user interaction in step (3), the user is allowed to control the video interpolation result by dragging, that is, the trajectory of the key point in the dragging process is obtained and converted into a Gaussian heat map, and then the Gaussian heat map is input into the encoding module for encoding to obtain the features of the key point trajectory, and then the features are injected into the 3D U-Net of the diffusion model.

4. The controllable generative video frame insertion method based on diffusion model according to claim 1, characterized in that: In step (4), the matching information of key points is obtained between the first frame and the last frame provided by the user through the feature point matching algorithm, and any key point pair that matches between the first frame and the last frame is recorded as p 0 and p n , the corresponding trajectory is obtained by interpolation estimation, and then all the estimated trajectories are obtained by traversing other matching key point pairs.

5. The controllable generative video frame insertion method based on diffusion model according to claim 4, characterized in that: The automated point trajectory tracking in step (4) is to update the point coordinates of the trajectory in the intermediate frame by using the similarity between the features in the diffusion model 3DU-Net for the trajectory generated by the feature point matching interpolation. Specifically, the trajectory generated by the feature point matching interpolation is first converted into a Gaussian heat map and then encoded to obtain the trajectory features, and then the trajectory features are injected into the diffusion model 3DU-Net; then, the features of the second-to-last upsampling module in the 3D U-Net are used to obtain the image features of each frame through bilateral linear interpolation; for any trajectory point k on the intermediate frame, a point set Ω whose distance to the trajectory point k is less than a threshold value r1 is obtained by searching; and then a nearest neighbor algorithm is used to calculate from the point set Ω the distance to the key point p of the first frame. 0 The point p closest to the feature k,0 And the key point p of the last frame n The point p closest to the feature k,n , if point p k,0 With p k,n If the distance is less than the threshold r2, the coordinate of the trajectory point k is updated to point p k,0 With p k,n The midpoint coordinates between , and for each round of 3D U-Net denoising, the trajectory points of the intermediate frames are updated once according to the above operations, and the updated trajectory features are used as the input of the next round of 3D U-Net denoising.

6. A computer device comprising a memory and a processor, characterized in that: The memory stores a computer program, and the processor is used to execute the computer program to implement a controllable generative video frame insertion method based on a diffusion model as claimed in any one of claims 1 to 5.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the controllable generative video frame insertion method based on a diffusion model as claimed in any one of claims 1 to 5 is implemented.

Images