A training-free video editing method based on frequency enhancement and conflict adaptive model

Abstract

The application provides a kind of training-free video editing method based on frequency enhancement and conflict adaptive model, belongs to the technical field of text-driven video editing;To solve the technical problems that the existing video editing exists motion and background confusion, artifacts appear in high-frequency area, it is difficult to adapt to scene complexity change, the technical scheme adopted includes: selecting Warped Diffusion model as the basic model, selecting Warped Diffusion as the sampler based on, balance structure reservation and editing intensity;Frequency enhancement flow control model and adaptive conflict resolution model are used respectively to perform video editing task;The frequency mask generated by frequency enhancement flow control is multiplied with the adaptive mask generated by adaptive conflict resolution to obtain the final editing mask, which realizes accurate editing of dynamic area and structure reservation of background;The application is used for text-driven video editing.

Description

Technical Field

[0001] This invention provides a training-free video editing method based on a frequency enhancement and conflict adaptive model, belonging to the field of text-driven video editing technology. Background Technology

[0002] Text-driven video editing techniques (especially those without training or inverse processes) are widely used because they can efficiently achieve semantically guided video operations. These video editing methods usually rely on pre-trained language-image diffusion models, but they are prone to temporal-spatial inconsistencies and flickering problems when dealing with sparse data and highly dynamic scenes.

[0003] Existing video editing methods mitigate these problems by introducing attention control or additional consistency modules, but still struggle to maintain high fidelity and temporal continuity in complex scenes. Editing methods without inverse processes operate primarily in the spatiotemporal domain, ignoring frequency domain information. Furthermore, insufficient spatiotemporal and frequency domain modeling leads to motion-background confusion, causing artifacts in high-frequency regions (such as texture details). Additionally, these methods rely on fixed resolution strategies, making it difficult to adapt to changes in scene complexity due to differences in complexity between different videos or within the same video. Summary of the Invention

[0004] To address the technical problems existing in the background art, the present invention provides a training-free video editing method based on a frequency enhancement and conflict adaptive model, comprising the following video editing steps:

[0005] Step S1: Prepare for video editing, specifically including:

[0006] Step S1.1: Select the Warped Diffusion model as the base model, select the sampler based on Warped Diffusion, and balance the preservation of structure with the intensity of editing;

[0007] Step S1.2: Select multiple mainstream video editing methods and adopt the editing process and default parameters of each video editing method;

[0008] Step S1.3: Run all video editing experiments using a single test GPU;

[0009] Step S1.4: Set the test set to a mixture of internet videos and generated videos, and perform text-video editing tasks covering the various mainstream video editing methods selected in step S1.2;

[0010] Step S2: Perform video editing tasks using a frequency-enhanced flow control model, specifically including:

[0011] Step 2.1: Perform a short-time Fourier transform on the video frames to extract features such as local energy and spectral center;

[0012] Step 2.2: Calculate the frequency weighting map using the weighting formula. The formula is as follows:

[0013] ;

[0014] Among them, α, β, and γ are used to balance local energy, and δ and ζ control the weights of the spectral center and spectral bandwidth, respectively.

[0015] Step 2.3: Generate a frequency mask, the expression is:

[0016] ;

[0017] Smoothing and truncation of the weight map yields a mask that distinguishes between dynamic regions and static backgrounds;

[0018] Step S3: Perform video editing tasks using an adaptive collision resolution model, specifically including:

[0019] Step 3.1: Calculate the conflict diagram C. The calculation formula is:

[0020] C = ||Vsource - Vtarget||2;

[0021] The degree of editing conflict is quantified based on the L2 norm of the gradient velocity fields of the source and target videos;

[0022] Step 3.2: Calculate the multi-dimensional scene complexity score. The calculation formula is as follows:

[0023] ;

[0024] The scene complexity score is obtained by fusing texture complexity, motion complexity, and conflict graph C.

[0025] Step 3.3: Generate an adaptive mask, the expression is:

[0026] ;

[0027] An edit mask is generated that adapts to the complexity of the scene by using an exponential function and a truncation operation.

[0028] Step S4: Multiply and fuse the frequency mask generated by the frequency enhancement flow control model with the adaptive mask generated by the adaptive conflict resolution model. The expression is:

[0029] ;

[0030] The final edit mask is obtained, enabling precise editing of dynamic areas while preserving the structure of the background.

[0031] Step S5: Conduct ablation experiments on the frequency-enhanced flow control model and the adaptive conflict resolution model to verify the synergistic effect of each model;

[0032] Step S6: Quantitatively evaluate the video editing results, including calculating the following metrics:

[0033] Frame alignment accuracy, semantic accuracy, text-video or frame alignment, temporal consistency, and editing quality;

[0034] Qualitative evaluation of video editing results includes visualizing the results of different methods and analyzing the visual effects of background preservation, attribute modification accuracy, and temporal consistency.

[0035] The various mainstream video editing methods selected in step S1.2 are: FateZero, RAVE, VideoLDM, Text2Video-Zero, Prompt-to-Prompt, CrossAttentionControl, MotionCtrl, and StyleDrop.

[0036] The text-video editing task performed in step S1.4 specifically involves modifying object attributes and changing the scene.

[0037] The beneficial effects of this invention compared to existing technologies are as follows: The training-free video editing method based on frequency enhancement and adaptive conflict model provided by this invention mainly adopts training-free, frequency domain enhancement, and adaptive conflict control technologies. It can be quickly integrated into video editing tools or film and television post-production systems, and achieves local video editing through text commands. It can accurately separate dynamic targets from static backgrounds, overcome flicker and artifact defects in the video editing process, and its training-free feature is compatible with existing diffusion models. It requires no additional resources and is suitable for application scenarios such as multiple version iterations of advertisements, self-media creation, and compliance modifications. It effectively lowers the threshold for professional editing and has both high efficiency and stability. Attached Figure Description

[0038] The present invention will be further described below with reference to the accompanying drawings:

[0039] Figure 1 This is a flowchart illustrating the steps of the training-free video editing method based on a frequency enhancement and conflict adaptive model of the present invention. Detailed Implementation

[0040] The present invention provides a training-free video editing method based on frequency enhancement and adaptive conflict resolution models. Specifically, it is based on a training-free, goal-based editing framework and introduces two collaborative modules, namely the frequency enhancement flow control module (FFEC) and the adaptive conflict resolution module (ACR), to perform text-driven video editing tasks. This method achieves better semantic alignment, temporal stability and structural fidelity in complex scenes and is suitable for tasks such as video editing and denoising.

[0041] like Figure 1 As shown, to achieve the above objectives, the technical solution adopted by the present invention includes:

[0042] Step S1: Experimental Preparation:

[0043] Step S1.1: Base Model Selection: The Warped Diffusion model is used as the base model, and a sampler based on Warped Diffusion is selected to balance the preservation of structure and the intensity of editing.

[0044] Step S1.2: Baseline methods and parameters: Select 8 mainstream video editing methods: FateZero, RAVE, VideoLDM, Text2Video-Zero, Prompt-to-Prompt, CrossAttentionControl, MotionCtrl, and StyleDrop, and adopt the editing workflow and default parameters of each video editing method;

[0045] Step S1.3: Hardware and Environment: Run all experiments using a single NVIDIA RTX4090 GPU;

[0046] Step S1.4: Test data: The test set is a mixture of "Internet video + generative video" data, covering 8 types of text-video editing tasks (such as object attribute modification, scene transformation, etc.).

[0047] The following section describes the experimental implementation of the core module:

[0048] Step S2: Frequency Enhancement Flow Control Module (FFEC) Steps:

[0049] Step 2.1: Perform a short-time Fourier transform (STFT) on the video frames to extract features such as local energy (Eiow / Eid / Enigh) and spectral center;

[0050] Step 2.2: Calculate the frequency weighting graph using the weighting formula:

[0051] ;

[0052] Among them, α, β, and γ are used to balance local energy, and δ and ζ control the weights of the spectral center and spectral bandwidth, respectively.

[0053] Step 2.3: Generate a frequency mask:

[0054] ;

[0055] Smoothing and truncation of the weight map yields a mask that distinguishes between dynamic regions (high weight) and static background (low weight).

[0056] Step S3: Adaptive Collision Resolution Module (ACR) Steps:

[0057] Step 3.1: Calculate the conflict diagram C. The calculation formula is:

[0058] C = ||Vsource - Vtarget||2;

[0059] The degree of editing conflict is quantified based on the L2 norm of the gradient velocity fields of the source and target videos;

[0060] Step 3.2: Generate a multi-dimensional scene complexity score:

[0061] ;

[0062] Fusion texture complexity ( ), motion complexity ( The scene complexity score is obtained by combining the conflict graph C with the scene complexity graph C.

[0063] Step 3.3: Generate an adaptive mask:

[0064] ;

[0065] An edit mask is generated by using an exponential function and a truncation operation to adaptively adjust to the complexity of the scene.

[0066] Step S4: Multiply and fuse the frequency mask of FFEC with the adaptive mask of ACR.

[0067] ;

[0068] The final edit mask is obtained, enabling precise editing of dynamic areas while preserving the structure of the background.

[0069] Step S5: Perform the ablation experiment:

[0070] Group 1: Using only the FFEC model (frequency only);

[0071] Group 2: ACR model only (adaptive only);

[0072] Group 3: Simultaneous use of FFEC+ACR models (enhanced fusion);

[0073] The video editing effects of the three groups were compared to verify the synergistic effect of each model.

[0074] Step S6: Result Evaluation:

[0075] Quantitative evaluation: Six metrics are calculated: Frame-Acc (frame alignment accuracy), PickScore (semantic accuracy), CLIP-T / F (text-video / frame alignment), WarpSIM (temporal consistency), and QaE (editing quality).

[0076] Qualitative evaluation: Visualize the editing results of different methods and analyze visual effects such as background preservation, accuracy of attribute modification, and time consistency.

[0077] In the embodiments of the present invention, eight mainstream video editing methods were specifically used for evaluation experiments, and the experimental comparison data are shown in Table 1 below:

[0078] Table 1 Comparison of experimental results

[0079]

[0080] Based on the above evaluation experiments, it can be demonstrated that the video editing method provided in this application achieves an average improvement of +4.8% in CLIP-F and +3.9% in Frame-Acc on eight mainstream benchmark methods.

[0081] Table 2 below shows the data obtained from the ablation experiment:

[0082] Table 2 Ablation Experiment Results

[0083]

[0084] Stability: Within the video duration range of 5-30 seconds, CLIP-F fluctuation ≤ ±0.012, Frame-Acc fluctuation ≤ ±0.015; within the resolution range of 720P-4K, the core indicator fluctuation ≤ ±0.02, with no significant increase in artifacts;

[0085] Generality: The average CLIP-F for 6 types of scenes and 8 types of editing tasks is ≥0.735 and Frame-Acc is ≥0.820. Among them, the performance of human movement and natural landscape scenes is the best (CLIP-F is 0.765 and 0.758 respectively).

[0086] Based on the above experimental results, the present invention, through the synergistic effect of FFEC and ACR modules, significantly outperforms existing technologies in text-frame alignment, inter-frame stability, and artifact control, and also possesses good stability and scene adaptability.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training-free video editing method based on a frequency enhancement and conflict adaptive model, characterized in that: The video editing steps include the following: Step S1: Prepare for video editing, specifically including: Step S1.1: Select the Warped Diffusion model as the base model, select the sampler based on Warped Diffusion, and balance the preservation of structure with the intensity of editing; Step S1.2: Select multiple mainstream video editing methods and adopt the editing process and default parameters of each video editing method; Step S1.3: Run all video editing experiments using a single test GPU; Step S1.4: Set the test set to a mixture of internet videos and generated videos, and perform text-video editing tasks covering the various mainstream video editing methods selected in step S1.2; Step S2: Perform video editing tasks using a frequency-enhanced flow control model, specifically including: Step 2.1: Perform a short-time Fourier transform on the video frames to extract local energy and spectral center features; Step 2.2: Calculate the frequency weighting map using the weighting formula. The formula is as follows: ； Among them, α, β, and γ are used to balance local energy, and δ and ζ control the weights of the spectral center and spectral bandwidth, respectively. Step 2.3: Generate a frequency mask, the expression is: ； Smoothing and truncation of the weight map yields a mask that distinguishes between dynamic regions and static backgrounds; Step S3: Perform video editing tasks using an adaptive collision resolution model, specifically including: Step 3.1: Calculate the conflict diagram C. The calculation formula is: C = ||Vsource - Vtarget||2; The degree of editing conflict is quantified based on the L2 norm of the gradient velocity fields of the source and target videos; Step 3.2: Calculate the multi-dimensional scene complexity score. The calculation formula is as follows: ； The scene complexity score is obtained by fusing texture complexity, motion complexity, and conflict graph C. Step 3.3: Generate an adaptive mask, the expression is: ； An edit mask is generated that adapts to the complexity of the scene by using an exponential function and a truncation operation. Step S4: Multiply and fuse the frequency mask generated by the frequency enhancement flow control model with the adaptive mask generated by the adaptive conflict resolution model. The expression is: ； The final edit mask is obtained, enabling precise editing of dynamic areas while preserving the structure of the background. Step S5: Conduct ablation experiments on the frequency-enhanced flow control model and the adaptive conflict resolution model to verify the synergistic effect of each model; Step S6: Quantitatively evaluate the video editing results, including calculating the following metrics: Frame alignment accuracy, semantic accuracy, text-video or frame alignment, temporal consistency, and editing quality; Qualitative evaluation of video editing results includes visualizing the results of different methods and analyzing the visual effects of background preservation, attribute modification accuracy, and temporal consistency.

2. The training-free video editing method based on frequency enhancement and conflict adaptive model according to claim 1, characterized in that: The various mainstream video editing methods selected in step S1.2 are: FateZero, RAVE, VideoLDM, Text2Video-Zero, Prompt-to-Prompt, CrossAttentionControl, MotionCtrl, and StyleDrop.

3. The training-free video editing method based on a frequency enhancement and conflict adaptive model according to claim 1, characterized in that: The text-video editing task performed in step S1.4 specifically involves modifying object attributes and changing the scene.

Images