Video Editing Timing Consistency Enhancement Method and System

Abstract

This invention relates to the field of video editing technology and discloses a method and system for enhancing temporal consistency in video editing. The method includes: processing the original video frame sequence in the original video stream data using an image diffusion model to obtain a set of frame-by-frame editing results; processing the set of frame-by-frame editing results using both the image diffusion model and the video diffusion model to obtain the frame-by-frame latent variables output by the image diffusion model and the video latent variables output by the video diffusion model, and performing feature processing to obtain initial noise for denoising; inputting the initial noise into the video diffusion model to obtain approximate video frames; calculating the optical flow field between adjacent frames of the approximate video frames using a pre-trained optical flow network; determining the fusion latent variables based on the optical flow field; and decoding the fusion latent variables to obtain the final edited video frame sequence. This invention solves the problems of temporal instability and low processing efficiency in existing temporal consistency enhancement methods by using a multi-agent collaborative mechanism to perform closed-loop organization and iterative optimization of the generation, evaluation, and parameter adjustment processes.

Description

Technical Field

[0001] This invention relates to the field of video editing technology, and in particular to a method and system for enhancing the temporal consistency of video editing. Background Technology

[0002] With the rapid development of generative methods such as diffusion models, text-driven video editing has become a highly anticipated research area, significantly reducing content production costs in scenarios such as film and television production, advertising content generation, and virtual / digital human creation. However, compared to single-frame image editing, video editing requires simultaneous satisfaction of semantic instructions and visual coherence across consecutive frames. Therefore, in actual generation, problems such as cross-frame flickering and appearance drift are still common, resulting in insufficient temporal consistency and difficulty in consistently outputting high-quality results.

[0003] To address the aforementioned issues, researchers have proposed several improvement schemes, primarily including: temporal modeling enhancement, constraint and alignment, and staged guidance. Regarding temporal modeling enhancement, some works attempt to strengthen the modeling ability of temporal dependencies by introducing structures such as 3D convolution, temporal attention, or cross-frame feature propagation into the video diffusion network. In terms of constraint and alignment, some methods introduce constraints such as optical flow consistency, inter-frame reprojection, or temporal regularization, combined with structured conditions such as pose, depth, or edges, to restrict changes in adjacent frames, thus mitigating the structure drift problem to some extent. For staged guidance, the high-quality edited result of the first frame is typically used as an appearance reference, and its information is propagated to subsequent frames. Latent variable propagation or interpolation smoothing techniques are used to maintain cross-frame correlation during the diffusion generation process. Among these improvement schemes, a common approach is to combine the Image Diffusion Model (IDM) and the Video Diffusion Model (VDM) and introduce them into the video editing workflow to improve the temporal consistency of the generated results. These methods typically begin by using an existing image diffusion model to generate edited results that satisfy textual instructions frame by frame in a video stream. Then, a video diffusion model denoises the latent variables across multiple frames to compensate for cross-frame temporal inconsistencies and generate the final video. However, these methods still have significant limitations in practical applications: when the image and video diffusion models work together, their latent representations focus on spatial appearance and temporal dynamics respectively, resulting in significant differences in their distribution characteristics. Without effective alignment and collaborative processing methods, conflicts can easily arise during information transmission, leading to cross-frame temporal instability. Furthermore, at the system level, these methods usually employ a fixed single-channel generation process, lacking effective state feedback and collaborative adjustment mechanisms between functional modules. This makes it difficult to dynamically evaluate and adaptively adjust parameters based on the generated results, and rarely incorporates user subjective evaluation into a unified optimization framework. This often requires users to manually adjust parameters repeatedly and attempt multiple times, resulting in low overall interaction efficiency and a poor user experience. Therefore, existing temporal consistency enhancement methods suffer from temporal instability and low processing efficiency. Summary of the Invention

[0004] This invention provides a method and system for enhancing the timing consistency of video editing, in order to solve the problems of timing instability and low processing efficiency in existing timing consistency enhancement methods.

[0005] To achieve the above objectives, the present invention employs the following technical solution:

[0006] In a first aspect, the present invention provides a method for enhancing the temporal consistency of video editing, comprising: S1. Obtain the raw video stream data, and use the image diffusion model to process the raw video stream data to obtain a set of frame-by-frame editing results. ; S2, the set of frame-by-frame editing results Encode the data and then use the image diffusion model and video diffusion model for inversion to obtain the frame-by-frame latent variables on the image side. Video latent variables on the video side ; S3, the frame-by-frame latent variables on the image side Cross-modal alignment is performed to obtain image-side features. The image-side features Aligned image latent variables obtained by mapping to the video-side latent space ; S4. Regarding the video latent variables Latent variables of the aligned image Weighted fusion is performed to obtain mixed latent variables. and the mixed latent variables Inversion is performed to obtain the initial noise. ; S5, the initial noise The denoising step is performed using the input of the video diffusion model, and the output is an approximate video frame. It also calls a pre-trained optical flow network to calculate approximate video frames. Optical flow field between adjacent frames ; S6, Based on the optical flow field Identify latent variables for fusion and the fusion latent variables Decoding yields the final edited video frame sequence with enhanced temporal consistency. .

[0007] Optionally, in step S1, an image diffusion model is invoked to process the original video stream data to obtain a set of frame-by-frame editing results. include: Based on user needs, text prompts are determined, and an image diffusion model is invoked to analyze the original video frame sequence in the original video stream data according to the text prompts. Perform frame-by-frame editing to obtain a set of frame-by-frame edit results. The frame-by-frame editing process satisfies the following relationship: ; In the formula, This indicates the result of frame-by-frame editing. This indicates the image diffusion model being called. Indicates the original video frame. Indicates text prompt words, This indicates optional structured conditions in frame-by-frame editing. This indicates an adjustable set of generation parameters.

[0008] Optionally, in S2, the set of frame-by-frame editing results is... Encode the data and then use the image diffusion model and video diffusion model for inversion to obtain the frame-by-frame latent variables on the image side. Video latent variables on the video side include: Collection of frame-by-frame editing results The latent variables are obtained by inputting them into the VAE encoder. ; Set the number of inversion steps N inv The encoded latent variables The inversion is performed by inputting the data into the image diffusion model and the video diffusion model respectively. The intermediate diffusion step τ is then used to fuse the inversion results, yielding the frame-by-frame latent variables output by the image diffusion model. Video latent variables output by the video diffusion model The intermediate diffusion step τ is located in the middle stage of the diffusion process, and preferably τ = N. inv / 2 or its neighborhood.

[0009] Optionally, in S3, the frame-by-frame latent variables on the image side... Cross-modal alignment is performed to obtain image-side features. The image-side features Aligned image latent variables obtained by mapping to the video-side latent space include: For each frame of latent variables and video latent variables Feature encoding is performed to obtain image-side features. and video-side features ; Construct a multi-head cross-attention network to integrate image-side features. Video-side features as query vectors As key and value vectors, they are input into a multi-head cross-attention network for cross-module alignment, resulting in aligned image-side features. ; For the image-side features Perform residual fusion and linear projection to integrate the image-side features. Mapping back to the latent space tensor yields the aligned image latent variables.

[0010] Optionally, the loss function of the multi-head cross-attention network satisfies the following relationship: ; In the formula, Represents the alignment loss function. Represents frame-by-frame latent variables. Representing latent variables in video, This represents the total number of frames in the video frame sequence used in the calculation.

[0011] Optionally, in S4, the video latent variables are... Latent variables of the aligned image Weighted fusion is performed to obtain mixed latent variables. and the mixed latent variables Inversion is performed to obtain the initial noise. ,include: Set preset blending weights to adjust the latent variables of the aligned image. With video latent variables Weighted fusion is performed according to preset mixing weights to obtain mixed latent variables. The weighted fusion satisfies the following relationship: ; In the formula, Represents mixed latent variables, Indicates the preset mixed weights. Representing latent variables in the image, Represents latent variables in video; Set the number of inversion steps N inv Mixed latent variables The input video diffusion model performs inversion from the intermediate diffusion step τ to complete N. inv - The initial noise used for denoising is obtained after the inversion step. The intermediate diffusion step τ is located in the middle stage of the diffusion process, and preferably τ = N. inv / 2 or its neighborhood.

[0012] Optionally, in step S5, the initial noise is... As input to the video diffusion model, the output approximates a video frame. It also calls a pre-trained optical flow network to calculate approximate video frames. Optical flow field between adjacent frames include: Set diffusion step index In the diffusion step index At that time, the initial noise Convert to initial latent variables and based on the diffusion step index Initial latent variables Update to the latent variables of the current diffusion step. ; The potential variables of the current diffusion step The UNet3D network in the input video diffusion model is used to predict the noise term, and the latent variables of the current diffusion step are sampled using the DDIM sampler based on the prediction results. Perform an update to obtain the updated diffusion step latent variables. ; Set a trigger period K, and adjust the updated diffusion step latent variables based on the trigger period K. Low-resolution decoding is performed to obtain approximate video frames. Wherein, the triggering period K is the interval diffusion step index between each low-resolution decoding. ; Call the pre-trained optical flow network to process the approximate video frames. Input into the pre-trained optical flow network to calculate approximate video frames. Optical flow field between adjacent frames .

[0013] Optionally, in S6, based on the optical flow field Identify latent variables for fusion and the fusion latent variables Decoding yields the final edited video frame sequence with enhanced temporal consistency. ,include: The optical flow field Resampling and scale matching are performed according to the resolution ratio to obtain the latent variables of the current updated diffusion step. A resampled optical flow field with consistent spatial resolution is obtained, and this resampled optical flow field is mapped to the latent space to obtain the latent space optical flow. ; Based on latent space optical flow Warp-align the latent variables from the previous frame to obtain the aligned latent variables. The warp alignment satisfies the following relationship: ; In the formula, Represents the latent variables after alignment. Indicates the latent variables of the previous frame. Represents latent space optical flow; The aligned latent variables are subjected to an exponential moving average. Weigh and fuse the latent variables of the current frame to obtain the fused latent variables. The weighted fusion satisfies the following relationship: ; In the formula, This indicates the fusion of latent variables. Represents the time-series smoothing coefficient; Set the number of noise reduction steps N den The fused latent variables are used as new inputs to replace the initial noise. Repeat steps S5 and S6, and after each execution of steps S5 and S6, use the fused latent variables from the current round as the input for the next denoising step, until the number of denoising steps satisfies the denoising step number N. den The final edited video frame sequence is obtained by decoding the fusion latent variables output from the last denoising step. .

[0014] Optionally, the method further includes: S7. Determine the final edited video frame sequence. The quality evaluation index m is used to evaluate the final edited video frame sequence. A diagnosis is performed to obtain a diagnostic result; The quality evaluation indicators include: semantic consistency indicators, visual quality indicators, and temporal consistency indicators. The diagnostic results include: when the semantic consistency index, visual quality index, and temporal consistency index are all equal to or higher than the corresponding demand thresholds, the diagnostic result is determined to be that the demand is met; when the semantic consistency index, temporal consistency index, or any one of the temporal consistency indexes is lower than the demand threshold, the diagnostic result is determined to be that the demand is not met. S8. Construct a multilayer perceptron. When the diagnostic result meets the requirements, output the final edited video frame sequence. As the final edited video, when the diagnostic results do not meet the requirements, the quality evaluation index m, the current parameter set Θ, the historical editing records, and the diagnostic results are input into the multilayer perceptron to obtain the parameter adjustment set ΔΘ. The parameter adjustment set ΔΘ is then substituted into S1-S6 to update the adjustable generation parameter set. Preset mixed weights Triggering period K and timing smoothing coefficient Using the new current parameter set, S1-S6 are re-executed to obtain the updated final edited video frame sequence. The updated final edited video frame sequence is evaluated using the aforementioned quality evaluation index m. Perform the diagnosis until the results meet the requirements, then output the updated final edited video frame sequence. As the final edited video; Wherein, the current parameter set Θ is the adjustable generation parameter set. Preset mixed weights Triggering period K and timing smoothing coefficient The set of parameters; The set of parameter adjustment values ​​ΔΘ is the set of adjustable generated parameters. Adjustment amount ΔΘ1, preset mixed weight The adjustment amount ΔΘ2, the adjustment amount ΔΘ3 of the trigger period K, and the timing smoothing coefficient. The set of adjustment amounts ΔΘ4.

[0015] In a second aspect, embodiments of this application provide a video editing timing consistency enhancement system for performing the video editing timing consistency enhancement method as described in the second aspect, the system comprising: The frame-by-frame video generation module is used to construct a frame-by-frame video generation agent and to execute step S1 in the video editing temporal consistency enhancement method using the frame-by-frame video generation agent. The feature alignment and hybrid inversion module is used to construct the feature alignment and hybrid inversion agent and to perform steps S2-S4 in the video editing temporal consistency enhancement method using the feature alignment and hybrid inversion agent. The optical flow smoothing module is used to construct a temporal smoothing agent and to perform steps S5-S6 in the video editing temporal consistency enhancement method using the temporal smoothing agent. The Critics module is used to build a critic agent and utilize the critic agent to perform step S7 in the video editing timing consistency enhancement method; An iterative loop coordinator is used to execute step S8 in the video editing timing consistency enhancement method.

[0016] Beneficial effects: The video editing temporal consistency enhancement method provided by this invention aligns the frame-by-frame latent variables on the image side with the video latent variables on the video side, and uses a hybrid inversion method to invert the aligned latent variables to obtain initial noise, thereby improving the effectiveness and stability of the initial noise. This effective and stable initial noise is then used to denoise video frames, effectively avoiding noise interference with the temporal consistency enhancement process. Based on the frame-by-frame editing results generated by the image diffusion model, a video diffusion model is further introduced for processing, and optical flow fields are used to align and fuse adjacent frames, effectively suppressing common problems in video editing such as cross-frame flickering and appearance drift, thus enhancing the temporal consistency of the generated video. Building upon the high-quality frame-by-frame appearance provided by the image diffusion model, the video diffusion model models temporal dependencies, and an optical flow-guided fusion mechanism is introduced, achieving a balance between maintaining spatial details and temporal coherence, avoiding detail blurring or structural loss caused by solely relying on temporal modeling. By constructing an initial noise generation path from the frame-by-frame editing results to the video diffusion model, combined with the optical flow-guided latent variable fusion mechanism, this invention reduces the overhead of manual adjustment and repeated generation while ensuring video quality, improving the overall efficiency of video editing. Attached Figure Description

[0017] Figure 1 This is a flowchart of a preferred embodiment of the video editing timing consistency enhancement method of the present invention; Figure 2 This is a schematic diagram of the overall framework of the video editing timing consistency method according to a preferred embodiment of the present invention; Figure 3 A schematic diagram of the hybrid inversion model provided in a preferred embodiment of the present invention; Figure 4 A schematic diagram of a feature alignment model provided in a preferred embodiment of the present invention; Figure 5 This is a schematic diagram of an optical flow temporal smoothing model provided in a preferred embodiment of the present invention; Figure 6 This is a schematic diagram of a multi-agent collaborative architecture provided in a preferred embodiment of the present invention. Detailed Implementation

[0018] The technical solution of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms "an" or "a" and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms "connected" or "linked" and similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up," "down," "left," "right," etc., are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship also changes accordingly.

[0020] Example 1 In this example, the present invention proposes a method for enhancing the temporal consistency of video editing, such as... Figure 1-2 As shown. Specifically, it includes the following steps: S1) Frame-by-frame video generation step; S2-S4) Feature alignment and hybrid inversion step; S5-S6) Optical flow temporal smoothing step; S7-S8) Critics evaluation and parameter tuning step.

[0021] S1) The purpose of the frame-by-frame video generation step is to independently edit each frame based on the original video frame sequence, text prompts, and optional structured conditions, by calling the Image Diffusion Model (IDM) (ControlNet, T2I-Adapter, or InstructPix2Pix can be selected), and outputting a set of frame-by-frame edit results. , where T=8, provides the basic appearance result for subsequent feature alignment and temporal consistency optimization.

[0022] Frame-by-frame editing processes satisfy the following relationship: ; In the formula, This indicates the result of frame-by-frame editing. This indicates the image diffusion model being called. Indicates the original video frame. Indicates text prompt words, This indicates optional structured conditions in frame-by-frame editing. This indicates an adjustable set of generation parameters.

[0023] S2) Feature alignment and hybrid inversion step: This step is used to establish a controllable initialization bridge between the IDM and VDM, and to improve the effectiveness and stability of the initial noise. This step is as follows... Figure 3 As shown, the specific steps include: S21) VAE encoding step, which generates the frame-by-frame edited results from the frame-by-frame video generation step. Input the VAE encoder to obtain the corresponding latent variable sequence. .

[0024] S22) Latent variable acquisition step, the latent variables The IDM and VDM are input separately and inverted to a preset intermediate step τ, where the IDM and VDM are consistent with the IDM called in step S1 and the VDM called in step S3, respectively. The inversion is performed in a total of N=50 steps. τ=N / 2 (or its neighborhood) is preferably chosen as the fusion point to balance appearance detail preservation and temporal structure modeling, resulting in... Same dimension and .

[0025] S3) Feature alignment step, used to map the latent representation on the IDM side to the shared latent space on the VDM side, to reduce the distribution difference between the latent representations of the two and improve the temporal stability of subsequent video diffusion generation. This step is as follows: Figure 4 As shown, the specific steps include: S31) Feature encoding step, which encodes latent features and The image is fed into an encoder for feature encoding to obtain image-side features. and video-side features .

[0026] S32) Attention alignment step, which will align features As a query, As the key and value, the input is used to perform cross-modal alignment in a multi-head cross-attention module to obtain the aligned image-side features. .

[0027] S33) Residual fusion and projection step, which affects the cross-attention output. Perform residual fusion and linear projection to map them back to the latent space tensor form, obtaining the aligned IDM latent variables. .

[0028] (S34) Training steps: Samples are selected from the TIP-I2V dataset and uniformly cropped and scaled to the target resolution (512×512) and a fixed number of frames (8 frames). Inversion is performed on the reference images and video clips to extract latent feature pairs, which are used as training samples for the model. The loss function used for training is the alignment loss function, as shown in the following formula: ; In the formula, Represents the alignment loss function. Represents frame-by-frame latent variables. Representing latent variables in video, This represents the total number of frames in the video frame sequence used in the calculation.

[0029] S4) Weighted mixing step: The aligned IDM latent variables and VDM-side latent variables are mixed according to a preset mixing weight α. Weighted fusion is performed, with α ranging from 0.4 to 0.7, to obtain the mixed latent variables. The formula is as follows: ; In the formula, Represents mixed latent variables, Indicates the preset mixed weights. Representing latent variables in the image, This represents a latent variable in the video.

[0030] The remaining inversion steps use mixed latent variables. Continue performing the remaining inversion steps (i.e., N-) in VDM. This yields the initial noise used for subsequent VDM generation. .

[0031] S5) Optical flow temporal smoothing step, which enhances the temporal consistency of adjacent frames by periodically introducing optical flow constraints and latent space fusion mechanisms. This step is as follows: Figure 5 As shown, the specific steps include: S51) Preliminary denoising step, which involves removing the latent variables from the current diffusion step. The UNet3D network, input to VDM (VidRD or ZeroScope can be used), predicts the noise term, and performs an update through the DDIM sampler to obtain intermediate latent variables that are closer to the clean video. ,in This is the index for the diffusion step.

[0032] S52) Motion estimation step, according to a preset period at each interval Each diffusion step triggers a motion estimation, and the triggering period K can be 4 to 8: for the intermediate latent variables corresponding to each frame. Low-resolution decoding is performed to obtain approximate video frames. And use a pre-trained optical flow network (such as RAFT or GMFlow) to calculate the optical flow field between adjacent frames. .

[0033] S6) Based on the optical flow field Identify latent variables for fusion and the fusion latent variables Decoding yields the final edited video frame sequence with enhanced temporal consistency. .

[0034] S61) Spatial adjustment and mapping step, which adjusts the optical flow field Resampling and scale matching are performed according to the resolution ratio to ensure consistency with the spatial resolution of the current latent variables. The resampled optical flow is then mapped to the latent space to obtain the latent space optical flow. .

[0035] S62) Temporal alignment step: Based on the latent space optical flow, warp alignment is performed on the latent variables of the previous frame to obtain the aligned latent variables, as shown in the following formula: ; In the formula, Represents the latent variables after alignment. Indicates the latent variables of the previous frame. This represents the optical flow in latent space.

[0036] S63) The smoothing step uses an exponential moving average (EMA) to weight and fuse the latent variables of the current frame with the aligned latent variables of the previous frame in order to suppress flicker and enhance coherence. The formula is as follows: ; In the formula, This indicates the fusion of latent variables. Represents the time series smoothing coefficient. For time series smoothing coefficients, A value of 0.2 to 0.5 is acceptable.

[0037] (S64) In the subsequent denoising steps, the fused latent variables are used as input for the next diffusion step to continue iterating until all denoising steps are completed. The total number of denoising steps is N=50, and the final edited video frame sequence is obtained after decoding. .

[0038] The above steps complete the temporal consistency enhancement in a full video editing process, and the final output of the steps is the final edited video frame sequence. This represents the result of a complete timing consistency enhancement step. For simple video editing, a single timing consistency enhancement step can produce a satisfactory result. However, as video resolution continues to improve, the number of frames per second (fps) in the video increases. For high-resolution video streams with a high fps, a single timing consistency enhancement step may not meet the user's needs. Alternatively, the user may have specific requirements when generating the final video stream, making a single timing consistency enhancement step insufficient for their current needs.

[0039] To accommodate high-resolution video streams with high frame rates per second and meet specific user needs, this embodiment introduces an evaluation, feedback, and regeneration step, forming a closed-loop optimized video editing workflow of "generation—evaluation—feedback—regeneration," thereby improving user interaction efficiency and user experience. As a preferred method in this embodiment, the evaluation, feedback, and regeneration specifically include the following steps: S7) Critics Evaluation and Parameter Tuning Step: This step is used to quantitatively evaluate the currently generated video results, diagnose problems, and output parameter update suggestions. This step specifically includes the following steps: S71) Indicator calculation step, calculate the quality evaluation index vector m for the generated video, the index including at least: semantic consistency index (Clip-Score-Text), visual quality index (Clip-Score), temporal consistency index (FVD, SSIM), etc.

[0040] S72) Problem diagnosis step: Based on the index vector m, perform problem judgment and output diagnosis results: For example, when Clip-Score-Text is below the threshold and FVD does not deteriorate significantly, it is judged as insufficient semantic consistency; when FVD increases significantly and SSIM decreases at the same time, it is judged as temporal instability; when SSIM decreases and Clip-Score decreases synchronously, and FVD does not change much, it is judged as loss of structural details or excessive smoothing, etc.

[0041] S8) Parameter tuning strategy generation step, which uses a multilayer perceptron MLP (three-layer fully connected network), with the input being the indicator vector m, historical records, the current parameter set Θ and the diagnostic results, and the output being the parameter adjustment amount ΔΘ.

[0042] Wherein, the current parameter set Θ is the adjustable generation parameter set. Preset mixed weights Triggering period K and timing smoothing coefficient The set of parameters; The set of parameter adjustment values ​​ΔΘ is the set of adjustable generated parameters. Adjustment amount ΔΘ1, preset mixed weight The adjustment amount ΔΘ2, the adjustment amount ΔΘ3 of the trigger period K, and the timing smoothing coefficient. The set of adjustment amounts ΔΘ4.

[0043] The above steps constitute the entire process of constructing the video editing timing consistency method of this invention.

[0044] Example 2 For the well-constructed video editing timing consistency method, the AutoGen agent framework is used to realize the organization, orchestration, communication, and collaboration among agents, thereby building a multi-agent collaborative framework, such as... Figure 6 As shown, the specific steps include: S1) Human-computer interaction and task initialization steps: The user inputs the original video as raw video stream data, inputs text commands P as text prompts, and optional condition information (such as pose / skeleton, depth, edges, etc.). The iterative loop coordinator parses and normalizes the user input, generating task configuration and global state information (including input data path, frame number / resolution, initial parameter set Θ, intermediate result storage location, etc.).

[0045] S2) The generation chain execution steps driven by the iterative loop coordinator, which is a finite state machine, sequentially schedule the frame-by-frame video generation agent, the feature alignment and hybrid inversion agent, and the temporal smoothing agent to perform generation tasks according to a preset process. The frame-by-frame video generation agent is used to edit the original video frame by frame based on user input to obtain a set of frame-by-frame edit results; the feature alignment and hybrid inversion agent performs feature alignment based on the set of frame-by-frame edit results to obtain image latent variables, and then performs hybrid inversion on the image latent variables to obtain initial noise; the temporal smoothing agent can then perform a denoising step based on the initial noise and call a pre-trained optical flow network to calculate the optical flow field, thereby decoding the latent variables based on the optical flow field to obtain the final edited video frame sequence with enhanced temporal consistency. During the generation process, large-scale data (video frames, latent variables, etc.) are referenced through disk paths or cache IDs, while small-scale data (parameters, indicators, etc.) are transmitted in messages using JSON / dictionary structures; the iterative loop coordinator maintains and updates the global state to complete the "state-transferring" collaboration.

[0046] S3) The critic evaluation and coordination step, which involves a closed-loop retry process, is a crucial step in achieving evaluation and feedback. As a preferred step in this embodiment, it is an important evaluation and feedback mechanism for realizing the "generation-evaluation-feedback-regeneration" process. This involves evaluating the quality of the final generated result: when the requirements are not met, the iterative loop coordinator triggers the critic agent to perform a quantitative evaluation and problem diagnosis of the generated result, and outputs parameter update suggestions. The iterative loop coordinator updates the global state based on these parameter suggestions and reschedules the relevant generating agents to execute a new round of generation and evaluation. When the user feedback is satisfactory, the iteration ends and the final edited video is output, thus forming a closed-loop optimization process of "generation-evaluation-feedback-regeneration".

[0047] Example 3 This application also provides a video editing timing consistency enhancement system for performing the video editing timing consistency enhancement method as described in Embodiment 1. The system includes: The frame-by-frame video generation module is used to construct a frame-by-frame video generation agent and to execute step S1 in the video editing temporal consistency enhancement method using the frame-by-frame video generation agent. The feature alignment and hybrid inversion module is used to construct the feature alignment and hybrid inversion agent and to perform steps S2-S4 in the video editing temporal consistency enhancement method using the feature alignment and hybrid inversion agent. The optical flow smoothing module is used to construct a temporal smoothing agent and to perform steps S5-S6 in the video editing temporal consistency enhancement method using the temporal smoothing agent. The Critics module is used to build a critic agent and utilize the critic agent to perform step S7 in the video editing timing consistency enhancement method; An iterative loop coordinator is used to execute step S8 in the video editing timing consistency enhancement method.

[0048] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for enhancing temporal consistency in video editing, characterized in that, include: S1. Obtain the raw video stream data, and use the image diffusion model to process the raw video stream data to obtain a set of frame-by-frame editing results. ; S2, the set of frame-by-frame editing results Encode the data and then use the image diffusion model and video diffusion model for inversion to obtain the frame-by-frame latent variables on the image side. Video latent variables on the video side ; S3, the frame-by-frame latent variables on the image side Cross-modal alignment is performed to obtain image-side features. The image-side features Aligned image latent variables obtained by mapping to the video-side latent space ; S4. Regarding the video latent variables Latent variables of the aligned image Weighted fusion is performed to obtain mixed latent variables. and the mixed latent variables Inversion is performed to obtain the initial noise. ; S5, the initial noise The denoising step is performed using the input of the video diffusion model, and the output is an approximate video frame. It also calls a pre-trained optical flow network to calculate approximate video frames. Optical flow field between adjacent frames ; S6, Based on the optical flow field Identify latent variables for fusion and the fusion latent variables Decoding yields the final edited video frame sequence with enhanced temporal consistency. .

2. The video editing timing consistency enhancement method according to claim 1, characterized in that, In step S1, the image diffusion model is invoked to process the original video stream data to obtain a set of frame-by-frame editing results. ,include: Based on user needs, text prompts are determined, and an image diffusion model is invoked to analyze the original video frame sequence in the original video stream data according to the text prompts. Perform frame-by-frame editing to obtain a set of frame-by-frame edit results. The frame-by-frame editing process satisfies the following relationship: ； In the formula, This indicates the result of frame-by-frame editing. This indicates the image diffusion model being called. Indicates the original video frame. Indicates text prompt words, This indicates optional structured conditions in frame-by-frame editing. This indicates an adjustable set of generation parameters.

3. The video editing timing consistency enhancement method according to claim 1, characterized in that, In step S2, the set of frame-by-frame editing results is... Encode the data and then use the image diffusion model and video diffusion model for inversion to obtain the frame-by-frame latent variables on the image side. Video latent variables on the video side ,include: Collection of frame-by-frame editing results The latent variables are obtained by inputting them into the VAE encoder. ; Set the number of inversion steps N inv The encoded latent variables The inversion is performed by inputting the data into the image diffusion model and the video diffusion model respectively. The intermediate diffusion step τ is then used to fuse the inversion results, yielding the frame-by-frame latent variables output by the image diffusion model. Video latent variables output by the video diffusion model The intermediate diffusion step τ is located in the middle stage of the diffusion process, and preferably τ = N. inv / 2 or its neighborhood.

4. The video editing timing consistency enhancement method according to claim 1, characterized in that, In S3, the frame-by-frame latent variables on the image side... Cross-modal alignment is performed to obtain image-side features. The image-side features Aligned image latent variables obtained by mapping to the video-side latent space ,include: For each frame of latent variables and video latent variables Feature encoding is performed to obtain image-side features. and video-side features ; Construct a multi-head cross-attention network to integrate image-side features. Video-side features as query vectors As key and value vectors, they are input into a multi-head cross-attention network for cross-module alignment, resulting in aligned image-side features. ; For the image-side features Perform residual fusion and linear projection to integrate the image-side features. Mapping back to the latent space tensor yields the aligned image latent variables. ; The loss function of the multi-head cross-attention network satisfies the following relationship: ； In the formula, Represents the alignment loss function. Represents frame-by-frame latent variables. Representing latent variables in video, This represents the total number of frames in the video frame sequence used in the calculation.

5. The video editing timing consistency enhancement method according to claim 1, characterized in that, In step S4, the video latent variables... Latent variables of the aligned image Weighted fusion is performed to obtain mixed latent variables. and the mixed latent variables Inversion is performed to obtain the initial noise. ,include: Set preset blending weights to adjust the latent variables of the aligned image. With video latent variables Weighted fusion is performed according to preset mixing weights to obtain mixed latent variables. The weighted fusion satisfies the following relationship: ； In the formula, Represents mixed latent variables, Indicates the preset mixed weights. Representing latent variables in the image, Represents latent variables in video; Set the number of inversion steps N inv Mixed latent variables The input video diffusion model performs inversion from the intermediate diffusion step τ to complete N. inv - The initial noise used for denoising is obtained after the inversion step. The intermediate diffusion step τ is located in the middle stage of the diffusion process, and preferably τ = N. inv / 2 or its neighborhood.

6. The video editing timing consistency enhancement method according to claim 1, characterized in that, In step S5, the initial noise As input to the video diffusion model, the output approximates a video frame. It also calls a pre-trained optical flow network to calculate approximate video frames. Optical flow field between adjacent frames ,include: Set diffusion step index In the diffusion step index At that time, the initial noise Convert to initial latent variables and based on the diffusion step index Initial latent variables Update to the latent variables of the current diffusion step. ; The potential variables of the current diffusion step The UNet3D network in the input video diffusion model is used to predict the noise term, and the latent variables of the current diffusion step are sampled using the DDIM sampler based on the prediction results. Perform an update to obtain the updated diffusion step latent variables. ; Set a trigger period K, and adjust the updated diffusion step latent variables based on the trigger period K. Low-resolution decoding is performed to obtain approximate video frames. Wherein, the triggering period K is the interval diffusion step index between each low-resolution decoding. ; Call the pre-trained optical flow network to process the approximate video frames. Input into the pre-trained optical flow network to calculate approximate video frames. Optical flow field between adjacent frames .

7. The video editing timing consistency enhancement method according to claim 1, characterized in that, In S6, based on the optical flow field Identify latent variables for fusion and the fusion latent variables Decoding yields the final edited video frame sequence with enhanced temporal consistency. ,include: The optical flow field Resampling and scale matching are performed according to the resolution ratio to obtain the latent variables of the current updated diffusion step. A resampled optical flow field with consistent spatial resolution is obtained, and this resampled optical flow field is mapped to the latent space to obtain the latent space optical flow. ; Based on latent space optical flow Warp-align the latent variables from the previous frame to obtain the aligned latent variables. The warp alignment satisfies the following relationship: ； In the formula, Represents the latent variables after alignment. Indicates the latent variables of the previous frame. Represents latent space optical flow; The aligned latent variables are subjected to an exponential moving average. Weigh and fuse the latent variables of the current frame to obtain the fused latent variables. The weighted fusion satisfies the following relationship: ； In the formula, This indicates the fusion of latent variables. Represents the time-series smoothing coefficient; Set the number of noise reduction steps N den The fused latent variables are used as new inputs to replace the initial noise. Repeat steps S5 and S6, and after each execution of steps S5 and S6, use the fused latent variables from the current round as the input for the next denoising step, until the number of denoising steps satisfies the denoising step number N. den The final edited video frame sequence is obtained by decoding the fusion latent variables output from the last denoising step. .

8. The video editing timing consistency enhancement method according to claims 1-7, characterized in that, The method further includes: S7. Determine the final edited video frame sequence. The quality evaluation index m is used to evaluate the final edited video frame sequence. A diagnosis is performed to obtain a diagnostic result; The quality evaluation indicators include: semantic consistency indicators, visual quality indicators, and temporal consistency indicators. The diagnostic results include: when the semantic consistency index, visual quality index, and temporal consistency index are all equal to or higher than the corresponding demand thresholds, the diagnostic result is determined to be that the demand is met; when the semantic consistency index, temporal consistency index, or any one of the temporal consistency indexes is lower than the demand threshold, the diagnostic result is determined to be that the demand is not met. S8. Construct a multilayer perceptron. When the diagnostic result meets the requirements, output the final edited video frame sequence. As the final edited video, when the diagnostic results do not meet the requirements, the quality evaluation index m, the current parameter set Θ, the historical editing records, and the diagnostic results are input into the multilayer perceptron to obtain the parameter adjustment set ΔΘ. The parameter adjustment set ΔΘ is then substituted into S1-S6 to update the adjustable generation parameter set. Preset mixed weights Triggering period K and timing smoothing coefficient Using the new current parameter set, S1-S6 are re-executed to obtain the updated final edited video frame sequence. The updated final edited video frame sequence is evaluated using the aforementioned quality evaluation index m. Perform the diagnosis until the results meet the requirements, then output the updated final edited video frame sequence. As the final edited video; Wherein, the current parameter set Θ is the adjustable generation parameter set. Preset mixed weights Triggering period K and timing smoothing coefficient The set of parameters; The set of parameter adjustment values ​​ΔΘ is the set of adjustable generated parameters. Adjustment amount ΔΘ1, preset mixed weight The adjustment amount ΔΘ2, the adjustment amount ΔΘ3 of the trigger period K, and the timing smoothing coefficient. The set of adjustment amounts ΔΘ4.

9. A video editing timing consistency enhancement system, used to execute the video editing timing consistency enhancement method as described in any one of claims 1-8, characterized in that, The system includes: The frame-by-frame video generation module is used to construct a frame-by-frame video generation agent and to execute step S1 in the video editing temporal consistency enhancement method using the frame-by-frame video generation agent. The feature alignment and hybrid inversion module is used to construct the feature alignment and hybrid inversion agent and to perform steps S2-S4 in the video editing temporal consistency enhancement method using the feature alignment and hybrid inversion agent. The optical flow smoothing module is used to construct a temporal smoothing agent and to perform steps S5-S6 in the video editing temporal consistency enhancement method.

10. The video editing timing consistency enhancement system according to claim 9, characterized in that, The system also includes: The Critics module is used to build a critic agent and utilize the critic agent to perform step S7 in the video editing timing consistency enhancement method; An iterative loop coordinator is used to execute step S8 in the video editing timing consistency enhancement method.

Images