Rectified model based generative video frame interpolation method and system

Abstract

The application discloses a rectification model-based generative video frame interpolation method and system, wherein the method comprises the following steps: S10, receiving the starting frame and the ending frame of a video as input, and receiving at least one of a text description, a motion trajectory and an intermediate frame as a control condition; S20, if the text description is included as the control condition, an adaptive classifier free guiding mechanism is used to dynamically adjust the text guiding strength according to the frame position; if the motion trajectory is included as the control condition, a Gaussian heat map is used to realize motion trajectory control, which is used for a user to customize the motion path of an object in the intermediate frame of the video; if the intermediate frame is included as the control condition, an intermediate frame feature fusion technology is used to mix the features of the intermediate frame specified by the user with a denoising latent representation; S30, the input is diffused in a latent space through a pre-trained multi-modal diffusion Transformer model; and S40, an intermediate frame sequence generated through the above processing is output to form a coherent video transition.

Description

Technical Field

[0001] This invention belongs to the field of video processing technology, specifically relating to a generative video frame interpolation method and system based on a rectification model. Background Technology

[0002] In the fields of multimedia and artificial intelligence, video processing technology, especially video frame interpolation, has made groundbreaking progress. Video frame interpolation, as one of the core tasks of computer vision, aims to achieve smooth transitions between consecutive video frames by synthesizing intermediate frames. Traditional methods typically treat it as a low-level visual task, assuming only minor motion between adjacent frames. Existing technical solutions mainly fall into two categories: methods based on optical flow estimation use motion vectors to guide frame generation; methods based on convolutional kernels directly synthesize pixels using spatially adaptive filters. The former is susceptible to motion estimation errors, while the latter is limited by the preset kernel size. To combine the advantages of both, some research attempts to construct end-to-end hybrid architectures.

[0003] In recent years, with the rise of large-scale video diffusion models, generative video frame interpolation methods have shown new possibilities. Representative works such as Latent Diffusion Model for VideoFrame Interpolation (LDM-VFI) transform the frame interpolation task into a conditional generation problem, leveraging diffusion models to improve visual quality; and Video Diffusion Model (VIDIM) employs a multi-level diffusion architecture to handle non-linear motion scenes. Despite these significant advancements, challenges remain when dealing with situations where there are substantial differences between the first and last frames. More importantly, existing solutions often focus on generating a single solution, lacking controllable design for the output.

[0004] In the field of text-to-video generation, pre-trained systems based on diffusion models have demonstrated outstanding generation capabilities. However, they still have shortcomings in fine-grained control: traditional control methods based on structural information (such as sketches and depth maps) are complex to operate and lack user-friendliness; while emerging motion trajectory control technologies provide more flexible operation space for video generation through intuitive interaction, significantly improving practical value.

[0005] Generative model-based methods have brought new possibilities to video frame interpolation. These methods treat frame interpolation as a conditional generation problem, using a latent diffusion model to generate interpolated frames with better visual quality. However, existing methods still have the following limitations: (1) they are mainly limited to video generation over short time spans; (2) they are difficult to handle significant motion differences between frames; (3) they lack fine control over the interpolation process, making it difficult to meet the needs of creative applications; and (4) the evaluation criteria are mainly aimed at traditional frame interpolation tasks and cannot fully reflect the creative potential of generative interpolation. Summary of the Invention

[0006] In view of the above-mentioned problems, the present invention provides a generative video frame interpolation method and system based on a rectification model, which improves the accuracy and controllability of video frame interpolation through innovative motion control strategies and optimized diffusion model architecture, so as to adapt to various complex motion scenarios and meet user customization needs, thereby promoting the further development of video processing technology.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0008] The first aspect of this invention provides a generative video frame interpolation method based on a rectification model, comprising the following steps:

[0009] S10, receive the start frame and end frame of the video as input, and at the same time receive at least one of the following as control conditions: text description, motion trajectory and intermediate frame.

[0010] S20, if text description is included as a control condition, an adaptive classifier free guidance mechanism is adopted to dynamically adjust the text guidance intensity according to the frame position;

[0011] If motion trajectory is included as a control condition, motion trajectory control is achieved through Gaussian heatmaps, which are used to define the motion path of objects in intermediate frames of the video for user customization.

[0012] If intermediate frames are included as control conditions, then intermediate frame feature fusion technology is used to mix the features of the user-specified intermediate frames with the denoising latent representation.

[0013] S30 uses a pre-trained multimodal diffusion Transformer model to diffuse the start frame, end frame, and input control conditions in the latent space.

[0014] S40, the intermediate frame sequence generated by the above processing outputs decodes the latent representation generated by the multimodal diffusion Transformer model into pixel space, forming a coherent video transition.

[0015] In one possible implementation, in S20, the adoption of an adaptive classifier free guidance mechanism to dynamically adjust the text guidance intensity according to the frame position includes: the adaptive classifier free guidance mechanism adopts a linear scheduling strategy, using stronger text guidance for intermediate frames and weaker text guidance for boundary frames.

[0016] In one possible implementation, the linear scheduling strategy includes adjusting the bootstrapping strength using a linear scheduling function:

[0017]

[0018] Where i is the frame index and n is the total number of frames, this mechanism reaches its maximum guidance strength w in the middle position. "ax The minimum boot strength w is reduced at the start and end frames. "$％ .

[0019] In one possible approach, in S10, the motion trajectory as input to the control condition includes a point trajectory input for the entire video length.

[0020] In one possible approach, in S10, the motion trajectory as input to the control conditions includes an incomplete trajectory input involving only a portion of the video frames.

[0021] In one possible approach, in S20, the use of intermediate frame feature fusion technology to mix the features of user-specified intermediate frames with the denoising latent representation includes:

[0022] Encode the intermediate frames into a latent representation;

[0023] Align its noise level at time step t;

[0024] The denoised latent representation is updated by weighted mixing.

[0025] The second aspect of this invention includes a generative video frame interpolation method and system based on a rectification model, comprising:

[0026] The input module is used to receive the start frame and end frame of the video as input, and at the same time receive at least one of the following as control conditions: text description, motion trajectory and intermediate frame.

[0027] The control module is used to dynamically adjust the text guidance intensity based on the frame position by employing an adaptive classifier free guidance mechanism if text description is included as a control condition.

[0028] If motion trajectory is included as a control condition, motion trajectory control is achieved through Gaussian heatmaps, which are used to define the motion path of objects in intermediate frames of the video for user customization.

[0029] If intermediate frames are included as control conditions, then intermediate frame feature fusion technology is used to mix the features of the user-specified intermediate frames with the denoising latent representation.

[0030] The diffusion processing module is used to diffuse the start frame, end frame, and input control conditions in the latent space using a pre-trained multimodal diffusion Transformer model.

[0031] The output module is used to output the intermediate frame sequence generated by the above processing, and decode the latent representation generated by the multimodal diffusion Transformer model into pixel space to form a coherent video transition.

[0032] In one possible implementation, the adaptive classifier free guidance mechanism dynamically adjusts the text guidance strength according to the frame position: the adaptive classifier free guidance mechanism adopts a linear scheduling strategy, using stronger text guidance for intermediate frames and weaker text guidance for boundary frames.

[0033] In one possible implementation, the linear scheduling strategy includes adjusting the bootstrapping strength using a linear scheduling function:

[0034]

[0035] Where i is the frame index and n is the total number of frames, this mechanism reaches its maximum guidance strength w in the middle position. "ax The minimum boot strength w is reduced at the start and end frames. "$％ .

[0036] The present invention has the following beneficial effects:

[0037] (1) The intermediate frame quality problem of generative models is solved by an adaptive guidance mechanism;

[0038] (2) The multimodal control system realizes multi-level creative control from semantics to pixels;

[0039] (3) The systematic evaluation benchmark provides a scientific evaluation standard for generative frame interpolation technology. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating the steps of a generative video frame interpolation method based on a rectification model according to an embodiment of the present invention.

[0041] Figure 2 This is a schematic diagram of the intermediate frame feature fusion mechanism in a specific application example of the present invention;

[0042] Figure 3 This is a schematic diagram illustrating the effect of a football rolling scene in a specific application example.

[0043] Figure 4 This is a schematic diagram of a generative video frame interpolation system based on a rectification model according to an embodiment of the present invention. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] See Figure 1 The diagram shows a flowchart of a generative video frame interpolation method based on a rectification model according to an embodiment of the present invention, including the following steps:

[0046] S10, receive the start frame and end frame of the video as input, and at the same time receive at least one of the following as control conditions: text description, motion trajectory and intermediate frame.

[0047] S20, if text description is included as a control condition, the Adaptive Classifier Free Guidance (AdaCFG) mechanism is adopted to dynamically adjust the text guidance intensity according to the frame position;

[0048] If motion trajectory is included as a control condition, motion trajectory control is achieved through Gaussian heatmaps, which are used to define the motion path of objects in intermediate frames of the video. That is, Gaussian heatmap encoding technology is used to convert the motion trajectory of key points drawn by the user on the input frame into a heatmap, which is then incorporated into the latent space through 1×1 convolution to meet the motion control requirements of different precision.

[0049] If intermediate frames are included as control conditions, then intermediate frame feature fusion technology is used to mix the features of the user-specified intermediate frames with the denoising latent representation.

[0050] S30 uses a pre-trained multimodal diffusion Transformer model to diffuse the start frame, end frame, and input control conditions in the latent space.

[0051] S40, the intermediate frame sequence generated by the above processing outputs decodes the latent representation generated by the multimodal diffusion Transformer model into pixel space, forming a coherent video transition.

[0052] In another embodiment of the present invention, a generative video frame interpolation method based on a rectification model is described. In S20, an adaptive classifier free guidance mechanism is adopted to dynamically adjust the text guidance strength according to the frame position. The adaptive classifier free guidance mechanism adopts a linear scheduling strategy, using stronger text guidance for intermediate frames and weaker text guidance for boundary frames.

[0053] Specifically, the linear scheduling strategy includes adjusting the bootstrapping strength using a linear scheduling function:

[0054]

[0055] Where i is the frame index and n is the total number of frames, this mechanism reaches its maximum guidance strength w in the middle position. "ax (Typical value is 3.0), decreasing to minimum boot strength w at the start and end frames. "$％ (Typical value is 1.0).

[0056] The text semantic control settings described above are based on a multimodal model pre-trained on large-scale image-text pairs, which has good image and text understanding capabilities and supports natural language descriptions to guide global content generation. For example, inputting "gradual transition from sunset to night scene" can guide a smooth transition in color style.

[0057] In another embodiment of the present invention, a generative video frame interpolation method based on a rectification model is provided. In S10, the motion trajectory as input as a control condition includes either a point trajectory input covering the entire video length or an incomplete trajectory input involving only a portion of the video frames. When the user connects the key points of the first and last frames, it is a complete trajectory; when the user only draws the trajectory starting from the first frame and only draws up to a certain middle frame without reaching the last frame, it is an incomplete trajectory.

[0058] In another embodiment of the present invention, a generative video frame interpolation method based on a rectification model, in step S20, the step of employing intermediate frame feature fusion technology to mix the features of user-specified intermediate frames with a denoising latent representation includes:

[0059] Encode the intermediate frames into a latent representation;

[0060] Align its noise level at time step t;

[0061] The denoised latent representation is updated by weighted mixing.

[0062] Furthermore, such as Figure 2 As shown, updating the denoised latent representation through weighted hybridization includes:

[0063]

[0064] The intermediate frame encoding can be potentially represented as: The mixing coefficient is s, and the latent features of the generated video are: The latent representation of the hybrid denoising is as follows

[0065] Another embodiment of the present invention provides a generative video frame interpolation method based on a rectified model. This method uses a pre-trained multimodal diffusion Transformer model to diffuse the input in the latent space. The reverse process utilizes the model for denoising, and the loss function is defined as follows:

[0066]

[0067] Where c txt For text conditions, z t For the potential representation of video, z / =(z 1 ,z ％ ) represents the aggregated potential features of the start and end frames. For mathematical expectation operator; v t Let be the conditional probability prediction function, representing the prediction of a given conditional information (c). txt ,z / Under these conditions, the model applies to variable z. t The prediction function. t The target vector or expected value; It is the square of the norm.

[0068] In a specific application example, such as Figure 3 As shown, the specific implementation process of the method of this embodiment of the invention is illustrated using the "football rolling" scenario as an example:

[0069] Enter the start frame (soccer ball on the left) and the end frame (soccer ball on the right). Optional text description: "Blue soccer ball rolling on the grass".

[0070] In this embodiment of the invention, the system automatically applies the AdaCFG mechanism, using stronger text guidance (e.g., w=3) for the generated intermediate frames (frames 3-5) and weaker guidance (e.g., w=1) for the frames near the boundary (frames 1-2 and 6-7).

[0071] To precisely control the trajectory of a football, the user can provide a series of trajectory points, which the system will convert into a Gaussian heatmap and incorporate into the generation process.

[0072] If a specific intermediate state is required (such as the moment a soccer ball is kicked), the user can provide an intermediate frame, and the system will use feature fusion technology to ensure that the generated sequence contains that state.

[0073] The system outputs a video sequence showing the smooth rolling of a soccer ball from left to right while maintaining a consistent background and natural motion. Without text control, the soccer ball exhibits noticeable and unreasonable deformation (the first row of the ball is stretched), and without AdaCFG, the ball shows significant motion blur. Using the method described in this embodiment of the invention yields a clearer and more reasonable result.

[0074] To address the shortcomings of existing evaluation systems, a benchmark dataset containing thousands of test samples has been constructed. It covers seven major application scenarios, including:

[0075] - Editing transitions: Includes local editing (adding and deleting objects) and global editing (style transfer).

[0076] -View transition: Acquired from multi-view 3D reconstruction data

[0077] - Cartoon in-between frames: covering traditional animation and modern CG styles

[0078] - Image warping: including object transformation and scene transition

[0079] -Time Shift: A Compressed Expression Showing Long-Term Changes

[0080] -Temporal super-resolution: the benchmark for traditional frame interpolation tasks

[0081] - Generalized scene transitions: Creative connections between arbitrary image pairs

[0082] The evaluation system includes eight quantitative indicators, which are evaluated automatically using a pre-trained model, including:

[0083] - Temporal consistency: Calculate topic similarity based on self-DIstillation with NO labels-Vision Transformer (DINO-ViT) and evaluate background consistency by contrastive language-image pre-training (CLIP);

[0084] - Motion quality: Recurrent All-Pairs Field Transforms (RAFT) optical flow analysis of dynamics, and AMT (All-Pairs Multi-Field Transforms) model evaluation of smoothness;

[0085] - Visual quality: LAION-5B aesthetic score (0-10) and MUSIQ (Multi-scale Image Quality Transformer) imaging quality (0-100);

[0086] -System metrics: ViCLIP calculates input / output consistency, and inter-frame difference measures time flicker;

[0087] Experimental results show that this invention significantly outperforms existing methods in creative frame interpolation tasks. In the generation of in-between frames for animations requiring precise control, user surveys indicate that most animators prefer the transition effects generated by this invention.

[0088] For the method implementation examples, see [link to relevant documentation]. Figure 4 The diagram shown is a principle block diagram of a generative video frame interpolation system based on a rectification model, according to another embodiment of the present invention, including:

[0089] The input module is used to receive the start frame and end frame of the video as input, and at the same time receive at least one of the following as control conditions: text description, motion trajectory and intermediate frame.

[0090] The control module is used to dynamically adjust the text guidance intensity based on the frame position by employing an adaptive classifier free guidance mechanism if text description is included as a control condition.

[0091] If motion trajectory is included as a control condition, motion trajectory control is achieved through Gaussian heatmaps, which are used to define the motion path of objects in intermediate frames of the video for user customization.

[0092] If intermediate frames are included as control conditions, then intermediate frame feature fusion technology is used to mix the features of the user-specified intermediate frames with the denoising latent representation.

[0093] The diffusion processing module is used to diffuse the start frame, end frame, and input control conditions in the latent space using a pre-trained multimodal diffusion Transformer model.

[0094] The output module is used to output the intermediate frame sequence generated by the above processing, and decode the latent representation generated by the multimodal diffusion Transformer model into pixel space to form a coherent video transition.

[0095] In another embodiment of the present invention, a generative video frame interpolation system based on a rectification model is provided. In S20, an adaptive classifier free guidance mechanism is adopted to dynamically adjust the text guidance strength according to the frame position. The adaptive classifier free guidance mechanism adopts a linear scheduling strategy, using stronger text guidance for intermediate frames and weaker text guidance for boundary frames.

[0096] Specifically, the linear scheduling strategy includes adjusting the bootstrapping strength using a linear scheduling function:

[0097]

[0098] Where i is the frame index and n is the total number of frames, this mechanism reaches its maximum guidance strength w in the middle position. "ax (Typical value is 3.0), decreasing to minimum boot strength w at the start and end frames. "$％ (Typical value is 1.0).

[0099] The text semantic control settings described above are based on a multimodal model pre-trained on large-scale image-text pairs, which has good image and text understanding capabilities and supports natural language descriptions to guide global content generation. For example, inputting "gradual transition from sunset to night scene" can guide a smooth transition in color style.

[0100] In another embodiment of the present invention, a generative video frame interpolation system based on a rectification model is provided. In the control module, the motion trajectory as a control condition input includes either a point trajectory input covering the entire video length or an incomplete trajectory input involving only a portion of the video frames. When the user connects the key points of the first and last frames, it is a complete trajectory; when the user only draws the trajectory starting from the first frame and only draws up to a certain intermediate frame without reaching the last frame, it is an incomplete trajectory.

[0101] In another embodiment of the present invention, a generative video frame interpolation system based on a rectification model, wherein the control module employs intermediate frame feature fusion technology to mix the features of user-specified intermediate frames with a denoising latent representation, comprising:

[0102] Encode the intermediate frames into a latent representation;

[0103] Align its noise level at time step t;

[0104] The denoised latent representation is updated by weighted mixing.

[0105] Furthermore, such as Figure 2 As shown, updating the denoised latent representation through weighted hybridization includes:

[0106]

[0107] The intermediate frame encoding can be potentially represented as: The mixing coefficient is s, and the latent features of the generated video are: The latent representation of the hybrid denoising is as follows

[0108] Another embodiment of the present invention provides a generative video frame interpolation system based on a rectification model. This system uses a pre-trained multimodal diffusion Transformer model to diffuse the input in the latent space. The reverse process utilizes the model for denoising, and the loss function is defined as follows:

[0109]

[0110] Where c txt For text conditions, z t For the potential representation of video, z / =(z 1 ,z ％ ) represents the aggregated potential features of the start and end frames. For mathematical expectation operator; v t Let be the conditional probability prediction function, representing the prediction of a given conditional information (c). txt ,z / Under these conditions, the model applies to variable z. t The prediction function. t The target vector or expected value; It is the square of the norm.

[0111] Those skilled in the art will understand that the beneficial effects of the method embodiments can also be applied to the system embodiments. The generative video interpolation system based on the rectification model, as described above, solves the intermediate frame quality problem of the generative model using an adaptive guidance mechanism; it achieves multi-level creative control from semantics to pixels using a multimodal control system; and it provides a scientific evaluation standard for generative interpolation technology using a systematic evaluation benchmark.

[0112] It should be understood that the exemplary embodiments described herein are illustrative and not restrictive. Although one or more embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit and scope of the invention as defined by the appended claims.

Claims

1. A rectified model-based generative video frame interpolation method, characterized in that, Includes the following steps: S10, receive the start frame and end frame of the video as input, and at the same time receive at least one of the text description and intermediate frames as control conditions. S20, if text description is included as a control condition, an adaptive classifier-based free guidance mechanism is adopted to dynamically adjust the text guidance strength according to the frame position. This includes: the adaptive classifier-based free guidance mechanism employs a linear scheduling strategy, using stronger text guidance for intermediate frames and weaker text guidance for boundary frames; the linear scheduling strategy includes adjusting the guidance strength using a linear scheduling function. where i is the frame index and n is the total number of frames, the mechanism reaches the maximum guiding strength at the middle position , at the start and end frames drop to the minimum guiding strength ; If intermediate frames are included as control conditions, then intermediate frame feature fusion technology is used to mix the features of the user-specified intermediate frames with the denoising latent representation, including: Encode the intermediate frames into a latent representation; Align its noise level at time step t; Updating the denoised latent representation through weighted mixing includes: wherein the intermediate frame encoding latent representation is , the mixing coefficient is s, the latent feature of the generated video is , and the mixed denoised latent representation is ; S30 uses a pre-trained multimodal diffusion Transformer model to diffuse the start frame, end frame, and input control conditions in the latent space. S40, the intermediate frame sequence generated by the above processing outputs decodes the latent representation generated by the multimodal diffusion Transformer model into pixel space, forming a coherent video transition.

2. The generative video frame interpolation method based on the rectification model as described in claim 1, characterized in that, It further includes receiving motion trajectories as control conditions and implementing motion trajectory control through Gaussian heatmaps, which is used for user-defined motion paths of objects in intermediate video frames.

3. The generative video frame interpolation method based on the rectification model as described in claim 2, characterized in that, The motion trajectory, as input to the control conditions, includes point trajectory input for the entire video length.

4. The generative video frame interpolation method based on the rectification model as described in claim 2, characterized in that, The motion trajectory, as input to the control conditions, includes incomplete trajectory inputs that only involve a portion of the video frames.

5. A generative video frame interpolation system based on a rectification model, characterized in that, include: The input module is used to receive the start frame and end frame of the video as input, and at the same time receive at least one of the text description and intermediate frames as control conditions. The control module, if text description is included as a control condition, employs an adaptive classifier-based free guidance mechanism to dynamically adjust the text guidance strength based on frame position. This includes: the adaptive classifier-based free guidance mechanism using a linear scheduling strategy, applying stronger text guidance to intermediate frames and weaker text guidance to boundary frames; the linear scheduling strategy includes adjusting the guidance strength using a linear scheduling function. Where i is the frame index and n is the total number of frames, this mechanism reaches maximum guidance strength in the middle position. The minimum boot strength is reduced at the start and end frames. ; If intermediate frames are included as control conditions, then intermediate frame feature fusion technology is used to mix the features of the user-specified intermediate frames with the denoising latent representation, including: Encode the intermediate frames into a latent representation; Align its noise level at time step t; Updating the denoised latent representation through weighted mixing includes: The intermediate frame encoding can be potentially represented as: The mixing coefficient is s, and the latent features of the generated video are: The denoising effect after mixing can be represented as follows: ; The diffusion processing module is used to diffuse the start frame, end frame, and input control conditions in the latent space using a pre-trained multimodal diffusion Transformer model. The output module is used to output the intermediate frame sequence generated by the above processing, and decode the latent representation generated by the multimodal diffusion Transformer model into pixel space to form a coherent video transition.

6. The generative video frame interpolation system based on the rectification model as described in claim 5, characterized in that, It further includes receiving motion trajectories as control conditions and implementing motion trajectory control through Gaussian heatmaps, which is used for user-defined motion paths of objects in intermediate video frames.

7. The generative video frame interpolation system based on the rectification model as described in claim 6, characterized in that, The motion trajectory, as input to the control conditions, includes point trajectory input for the entire video length.

8. The generative video frame interpolation system based on the rectification model as described in claim 6, characterized in that, The motion trajectory, as input to the control conditions, includes incomplete trajectory inputs that only involve a portion of the video frames.

Images