Video editing method and video editing apparatus based on interaction effect propagation

Abstract

A video editing method and device based on interaction effect propagation are disclosed. The video editing method comprises: receiving an input video and a user prompt word; generating an interaction-aware prompt word based on the user prompt word; identifying a set of key frames and generating scene interaction effect information corresponding to the key frames using a pre-trained multi-modal large language model based on the input video and the interaction-aware prompt word; extracting diffusion features of each frame in the input video using an image generation model; editing the key frames using the image generation model based on the scene interaction effect information to generate edited key frames; jointly editing the edited key frames using the image generation model to generate jointly edited key frames; and editing other frames of the input video using the image generation model based on features of the jointly edited key frames to generate an edited output video.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision, and more specifically, to a method and apparatus for editing video. Background Technology

[0002] With the development of social media, the demand for personalized video editing has increased exponentially. Traditional video editing methods require manual frame-by-frame processing, which is complex and time-consuming. In addition, video editing must ensure the visual quality of each frame while maintaining the logical coherence of the overall video.

[0003] In recent years, generative artificial intelligence (AI) has been widely used in the field of computer vision. Most video editing methods generate target videos based on text prompts. However, most editing methods based on video generation models still have limitations in maintaining the consistency of video footage. Summary of the Invention

[0004] The present invention is provided in a brief form to introduce the choice of concepts further described in the following detailed description. This summary is not intended to identify key or / or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0005] According to one aspect of this disclosure, a video editing method based on interactive effect propagation is provided. The video editing method includes: receiving an input video and user prompts, wherein the input video includes multiple frames, and the user prompts indicate text prompts that will modify visual elements in the input video; generating interactive awareness prompts based on the user prompts; identifying a set of keyframes using a pre-trained multimodal large language model based on the input video and the interactive awareness prompts, and generating scene interaction effect information corresponding to the keyframes; extracting diffusion features of each frame in the input video using an image generation model; editing the keyframes using the image generation model based on the scene interaction effect information to generate edited keyframes; jointly editing the edited keyframes using the image generation model to generate jointly edited keyframes; and editing other frames of the input video using the image generation model based on the features of the jointly edited keyframes to generate an edited output video.

[0006] The steps of generating interactive awareness prompts based on user prompts may include: generating interactive awareness prompts based on preset templates, wherein the interactive awareness prompts are used to fine-tune the multimodal large language model to guide the multimodal large language model to perceive the scene interaction effects present in the input video.

[0007] Scene interaction effect information may include: keyframe numbers, timeline information corresponding to the keyframes, and natural language descriptions of scene interactions.

[0008] The image generation model can be a neural network model based on a diffusion model. The steps of editing keyframes using the image generation model may include: at each time step of the interaction effect generation, randomly selecting a set of target keyframes from the set of keyframes; obtaining a noisy vector mapped to the latent space for each target keyframe; and using the image generation model to generate the edited keyframes based on the noisy vectors and scene interaction effect information.

[0009] The steps for jointly editing the edited keyframes may include: calculating the self-attention features of each edited keyframe, wherein the self-attention features include query features, key features, and value features; concatenating the key features of the edited keyframes to generate extended joint attention; and jointly editing each target keyframe based on the extended joint attention, wherein the feature vector of the jointly edited keyframe includes the result of fusing the semantic features of the set of target keyframes.

[0010] The steps for extracting the diffusion features of each frame in an input video using an image generation model may include: obtaining a noisy vector mapped to the latent space for each frame; and extracting feature vectors using a self-attention layer of the image generation model to obtain the diffusion features of each frame.

[0011] The diffusion features of each frame can be used to establish inter-frame feature correspondences.

[0012] The steps for editing other frames of an input video using an image generation model may include: obtaining the nearest previous target keyframe and the next target keyframe to the current frame being edited; establishing inter-frame feature correspondences between the current frame and the previous target keyframe, and between the current frame and the next target keyframe, based on diffusion features; generating a feature vector for the current frame based on the inter-frame feature correspondences and the feature vectors of the previous and next target keyframes; and generating the edited current frame based on the feature vector of the current frame.

[0013] The feature vector of the current frame can be calculated based on the weighted sum of the feature vectors of the previous target keyframe and the next target keyframe, where the weight value is inversely proportional to the time difference between the target keyframe and the current frame.

[0014] The video editing method may further include: recommending prompts to the user based on the input video when no prompts are received from the user.

[0015] According to one aspect of this disclosure, a video editing apparatus based on interactive effect propagation is provided. The video editing apparatus includes: a preprocessing module configured to: receive an input video and user prompts, wherein the input video includes multiple frames, and the user prompts indicate text prompts that will modify visual elements in the input video; generate interactive awareness prompts based on the user prompts; a video perception module configured to: identify a set of keyframes based on the input video and the interactive awareness prompts, and generate scene interaction effect information corresponding to the keyframes; a feature extraction module configured to: extract diffusion features of each frame in the input video; an interactive effect generation module configured to: edit the keyframes based on the scene interaction effect information to generate edited keyframes; a joint editing module configured to: jointly edit the edited keyframes to generate jointly edited keyframes; and an editing effect propagation and generation module configured to: edit other frames of the input video using an image generation model based on the features of the jointly edited keyframes to generate an edited output video.

[0016] The preprocessing module can be configured to generate interactive awareness prompts based on preset templates. These prompts are used to fine-tune the multimodal large language model to guide it in perceiving the scene interaction effects present in the input video.

[0017] Scene interaction effect information may include: keyframe numbers, timeline information corresponding to the keyframes, and natural language descriptions of scene interactions.

[0018] The interaction effect generation module can be configured to: randomly select a set of target keyframes from the set of keyframes at each time step of interaction effect generation; obtain a noisy vector mapped to the latent space for each target keyframe; and generate edited keyframes based on the noisy vector and scene interaction effect information.

[0019] The joint editing module can be configured to: calculate the self-attention features of each edited keyframe, wherein the self-attention features include query features, key features, and value features; concatenate the key features of the edited keyframes to generate extended joint attention; and perform joint editing on each target keyframe based on the extended joint attention, wherein the feature vector of the jointly edited keyframe includes the result of fusing the semantic features of the set of target keyframes.

[0020] The feature extraction module can be configured to: obtain a noisy vector mapped to the latent space for each frame; extract feature vectors using the self-attention layer of the image generation model to obtain the diffusion features of each frame, wherein the diffusion features of each frame are used to establish the feature correspondence between frames.

[0021] The editing effect propagation and generation module can be configured to: obtain the nearest previous target keyframe and the next target keyframe to the current frame being edited; establish inter-frame feature correspondences between the current frame and the previous target keyframe, and between the current frame and the next target keyframe, based on diffusion features; generate the feature vector of the current frame based on the inter-frame feature correspondences and the feature vectors of the previous and next target keyframes; and generate the edited current frame based on the feature vector of the current frame.

[0022] The feature vector of the current frame can be calculated based on the weighted sum of the feature vectors of the previous target keyframe and the next target keyframe, where the weight value is inversely proportional to the time difference between the target keyframe and the current frame.

[0023] The preprocessing module can also be configured to recommend prompts to the user based on the input video if no prompts are received from the user.

[0024] According to one aspect of this disclosure, a non-transitory computer-readable storage medium stores instructions that, when executed by a processor, cause the processor to perform a video editing method according to an example embodiment of this disclosure.

[0025] Further aspects and / or advantages of the inventive concept will be set forth in part in the description which follows, and in part will be obvious from the description and / or may be learned by practice of various exemplary embodiments. Attached Figure Description

[0026] The above and other objects, features and advantages of this disclosure will become clearer from the following detailed description taken in conjunction with the accompanying drawings.

[0027] Figure 1 This is a flowchart illustrating a video editing method based on interactive effect propagation according to an example embodiment of the present disclosure.

[0028] Figure 2 This is a diagram illustrating a template for interactively perceived prompts according to an example embodiment of the present disclosure.

[0029] Figure 3 This is a diagram illustrating an example of scene interaction effect information output by MLLM according to an exemplary embodiment of the present disclosure.

[0030] Figure 4 This is a diagram illustrating the process of extracting features according to an example embodiment of the present disclosure.

[0031] Figure 5 This is a diagram illustrating the processing of editing keyframes according to an example embodiment of the present disclosure.

[0032] Figure 6This is a diagram illustrating the process of propagating editing effects according to an example embodiment of the present disclosure.

[0033] Figure 7 This is a diagram illustrating a video editing process according to an example embodiment of the present disclosure.

[0034] Figure 8 This is a block diagram illustrating a video editing apparatus based on interactive effect propagation according to an example embodiment of the present disclosure.

[0035] Figures 9 to 11 This is an example illustrating a video editing method based on interactive effect propagation according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0036] The following detailed description is provided to assist the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, upon understanding the disclosure of this application, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, except for those that must occur in a specific order, but may be changed as will become clear upon understanding the disclosure of this application. Furthermore, for clarity and brevity, descriptions of features known in the art may be omitted.

[0037] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein have been provided merely to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein that will be clear upon understanding the disclosure of this application.

[0038] The structural or functional descriptions of the examples disclosed herein are intended for illustrative purposes only, and the examples may be implemented in various forms. The examples are not intended to be limiting, but rather to include various modifications, equivalents, and substitutions within the scope of the claims.

[0039] Although the terms “first” or “second” are used to describe various components, the components are not limited to the terms. These terms should only be used to distinguish one component from another. For example, within the scope of the claims based on the concept of this disclosure, a “first” component may be referred to as a “second” component, or similarly, a “second” component may be referred to as a “first” component.

[0040] It will be understood that when a component is referred to as being "connected to" another component, the component may be directly connected to or combined with the other component, or there may be an intermediate component.

[0041] As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. It should also be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of the stated features, integrals, steps, operations, elements, components, or combinations thereof, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof.

[0042] Unless otherwise defined, all terms used herein (including technical or scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the examples pertain. It will also be understood that, unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and shall not be interpreted in an idealized or overly formalized sense.

[0043] The examples will be described in detail below with reference to the accompanying drawings. Regarding the reference numerals assigned to the elements in the drawings, it should be noted that the same elements will be represented by the same reference numerals, and redundant descriptions will be omitted.

[0044] Before describing the embodiments of this disclosure, a brief introduction to the terminology used in this disclosure will be given.

[0045] A Multimodal Large Language Model (MLLM) is a large language model capable of simultaneously understanding and processing information from multiple modalities (e.g., text, images, video, audio, etc.). By mapping inputs from different modalities to a unified semantic space, MLLM enables cross-modal content understanding and generation.

[0046] Few-shot prompting refers to providing a small number of examples when inputting instructions to a large language model to guide the model in understanding the task requirements and enabling it to quickly adapt to new tasks.

[0047] Fine-tuning refers to further adjusting the model parameters using a small amount of specific task data on top of a pre-trained model that already has general capabilities, so that it can be adapted to a specific application scenario.

[0048] Diffusion models are generative models that progressively add Gaussian noise to data, reducing it to pure noise, and then learn to gradually remove noise to recover the original data. Diffusion models are widely used in image and video content generation tasks. Adding noise is the forward process of a diffusion model; in this step, Gaussian noise is gradually added to the data until it becomes pure noise. Denoising is the reverse process; in this step, the model learns to gradually remove noise to recover the original data, which is the core step in content generation. Image generation models based on diffusion models (e.g., Stable Diffusion) are widely used in video editing.

[0049] The diffusion space is a potential representation space in which data is progressively noise-added or denoised using a diffusion model. Within this space, the data undergoes a gradual transition from clear to noisy (or vice versa). The generative capability of the diffusion model is achieved by learning the changes in the data distribution within this space.

[0050] DDIM inversion is a process that leverages the deterministic properties of DDIM (Denoising Diffusion Implicit Model) to inversely add noise to a real image, backtracking to a noise representation in the latent space. After obtaining the initial noise corresponding to the original image through inversion, text prompts can be used to modify the noise and regenerate the image. This allows for localized modifications (e.g., replacing "pizza" with "bread") while preserving the overall structure of the original image (e.g., layout, background). DDIM inversion is a crucial step in text-guided image editing tasks. The input image x0 is transformed into a latent variable x that resembles Gaussian noise after the DDIM inversion step. T .

[0051] Keyframes are video frames that represent significant changes in motion, scene, or narrative logic. During video editing, keyframes can be edited first, and then the diffusion features of the edited keyframes can be propagated to non-keyframes based on inter-frame feature correspondences.

[0052] Inter-frame feature correspondence refers to the feature matching and association between adjacent frames of a video for the same or similar content (e.g., objects, regions). During video editing, inter-frame feature correspondence is used to ensure that the same content maintains temporal and logical consistency across different frames during editing or generation.

[0053] Figure 1 This is a flowchart illustrating a video editing method based on interactive effect propagation according to an example embodiment of the present disclosure.

[0054] In step S100, an input video and user prompts may be received. The input video may be a video that the user wishes to edit and may include multiple frames. The user may provide editing prompts to guide the image generation model to edit the input video to generate a video with the user's desired effect. In an example embodiment, the editing prompts may be textual hints indicating that visual elements in the input video will be modified.

[0055] In step S200, interactively perceptive prompts can be generated based on user prompts.

[0056] User prompts can predefine scene interaction concepts, such as "a person leaves footprints on the beach", "a cat leaves traces while walking on the grass", "a cat breaks a vase in the scene", etc.

[0057] In the example implementation, interactive awareness prompts can be generated based on a preset template. These prompts can be used to fine-tune a multimodal large language model (MLLM) to guide it in perceiving scene interactions within the input video. A few-shot prompting method can be employed in the interactive awareness prompts to fine-tune the MLLM for better results. For example, a small number of keyframes can be extracted from the input video to guide the MLLM in discovering more scenes and keyframes.

[0058] In step S300, based on the input video and interactive awareness prompts, a pre-trained multimodal large language model (MLLM) can be used to identify a set of keyframes and generate scene interaction effect information corresponding to the keyframes.

[0059] In step S400, the diffusion features of each frame in the input video can be extracted using an image generation model.

[0060] In an example embodiment, the image generation model can be a neural network model based on a diffusion model. The diffusion features extracted using the image generation model can be used to establish inter-frame feature correspondences.

[0061] In the example embodiment, a noisy vector mapped to the latent space for each frame can be obtained, and a feature vector is extracted using the self-attention layer of the image generation model to obtain the diffusion features of each frame.

[0062] In step S500, the keyframes can be edited using an image generation model based on the scene interaction effect information to generate edited keyframes.

[0063] In step S600, the edited keyframes can be jointly edited using an image generation model to generate jointly edited keyframes.

[0064] In step S700, based on the features of the jointly edited keyframes, other frames of the input video can be edited using an image generation model to generate an edited output video.

[0065] Figure 2 This is a diagram illustrating a template for interactively perceived prompts according to an example embodiment of the present disclosure.

[0066] Taking the user prompt "a cat leaves tracks on the grass" as an example, interactive prompts can be generated based on the following template.

[0067] "Scene interaction definition: {A cat walks on the grass, leaving tracks}. Please analyze the input keyframe images and perceive the possible scene interaction effects in this video. The user explicitly specifies the interaction to be implemented as: {A cat walks on the grass, leaving tracks}. Please output the interactive effect that the user explicitly specifies needs to perceive in this video, and output other related effects that may be triggered by this modification according to the definition, and output according to the following format:" { "interaction": [ { "timeline":"{xx:xx-xx:xx}", "key_frame_no":"{xx-xx}", "interaction_effects":"{interaction effects prompt}" } ] } Sample output is as follows: { "interaction": [ { "timeline":"00:07-00:12", "key_frame_no": "10-15", "interaction_effects":"The cat rolls in the grass and leaves the marks." } ] }

[0068] Figure 3 This is a diagram illustrating an example of scene interaction effect information output by MLLM according to an exemplary embodiment of the present disclosure.

[0069] The input video, the keyframe numbers extracted by the user, and the interactive awareness prompts can be input into MLLM, enabling MLLM to discover objects and scenes in the input video, as well as possible scene interactions between objects.

[0070] MLLM can output a three-element array containing keyframe numbers, the corresponding timeline, and a natural language description of the scene interactions, which can be used to guide subsequent image editing.

[0071] Figure 4 This is a diagram illustrating the process of extracting diffusion features according to an example embodiment of the present disclosure.

[0072] Reference Figure 4 This allows us to obtain the noisy vector x for each frame of the original input video. T Despite Figure 4 The diagram illustrates the use of DDIM inversion to obtain a noisy vector. x T However, this disclosure is not limited to this system, and various methods can be used to obtain noisy vectors. x T .

[0073] Obtain the noisy vector for each frame x T In the future, feature vectors can be extracted using the self-attention layer of the image generation model. ,in, i This indicates the frame number. In the example embodiment, when the image generation model has a Unet architecture, the feature vector of each frame can be extracted in the self-attention layer of the encoder in the Unet architecture. This serves as a diffusion feature for subsequently establishing inter-frame feature correspondences.

[0074] Figure 5 This is a diagram illustrating the processing of editing keyframes according to an example embodiment of the present disclosure.

[0075] Editing video using a diffusion-based image generation model can include multiple denoising steps, each corresponding to a time step t. At each time step t, a set of keyframes can be randomly selected from the keyframes of the input video. Based on the scene interaction effect information output by the MLLM, the image generation model adds corresponding scene interaction effects to this randomly selected set of keyframes.

[0076] In the example embodiment, at each time step t, a set of keyframes can be randomly selected from the set of keyframes identified by the MLLM for editing. The selected keyframes can be referred to as target keyframes. In the example embodiment, target keyframes can be selected at different frame intervals at each time step t. If the image generation model completes the video editing process over T time steps, to ensure the video editing effect, it should be ensured that each keyframe is selected as a target keyframe at least once in the T time steps.

[0077] After selecting the target keyframes, a noisy vector mapping each target keyframe to the latent space can be obtained. x T Image generation models based on noisy vectors can be used. x T The scene interaction effect information output by MLLM is used to generate edited keyframes.

[0078] In the example embodiment, an input video may include multiple scenes, and each scene has a corresponding set of keyframes. When editing the keyframes, the video can be segmented based on the scene, so that a set of keyframes selected at each time step corresponds to a scene.

[0079] To improve the consistency of video footage, after editing the selected target keyframes, the edited target keyframes can be jointly edited, and the diffusion characteristics of the set of target keyframes can be updated.

[0080] Self-attention features can be calculated for each edited keyframe. For example, query features can be obtained. Q Key features K Sum value characteristics V The self-attention feature. For example, the query matrix, key matrix, and value matrix can be used respectively based on the self-attention feature from the first... i Frame-extracted feature maps are used to obtain query features. Q i Key features K i Sum value characteristics V i .

[0081] Edited keyframes i Key features K i The attention is spliced ​​together to generate an extended joint attention.

[0082] Equation (1) below shows a computational example of extended joint attention.

[0083] (1)

[0084] In equation (1), iThis indicates the index of the frame currently being processed, and d represents the dimension of the vector (Q or K). k = { ,…, } This represents the set of indices of the currently selected set of target keyframes; that is, a total of k target keyframes are selected at the current time step t. , … , The key vectors of all target keyframes are concatenated along the sequence dimension to form a larger key matrix.

[0085] After calculating the extended joint attention, the target keyframes can be jointly edited again through attention operations to generate jointly edited keyframes, allowing multiple target keyframes to share the global appearance.

[0086] The set of feature vectors of the jointly edited keyframes can be represented as: ,in, Indicates from keyframe i ( Edited image Extracted features.

[0087] The feature vector of each keyframe after joint editing may include the result of fusing the semantic features of the target keyframes.

[0088] Next, based on the inter-frame feature correspondence, the feature vectors of the jointly edited keyframes can be... Propagate to other frames of the input video.

[0089] Figure 6 This is a diagram illustrating the process of propagating editing effects according to an example embodiment of the present disclosure.

[0090] When editing is required on frames in the input video other than the target keyframe, the nearest preceding and following target keyframes to the current frame being edited can be obtained. Based on the extracted diffusion features of each frame, inter-frame feature correspondences between the current frame and the preceding and following target keyframes can be established. Based on these inter-frame feature correspondences and the feature vectors of the preceding and following target keyframes, a feature vector for the current frame can be generated. The edited current frame is then generated based on this feature vector.

[0091] In each time step t during feature propagation in the image generation model, each video frame whose features need to be propagated and edited can be denoted as the current frame. i Find the current frame i Nearest past target keyframe i- and the current framei Nearest Future Keyframe i+ The position with the highest spatial similarity to the current frame is used to establish the current frame. i The feature correspondence between it and its adjacent target keyframes.

[0092] In an example embodiment, the diffusion features of each frame of the extracted input video can be used to find the nearest neighbor. For example, the nearest neighbor can be calculated using the following equation (2). The nearest neighbor can be represented as the location in the nearest past target keyframe and future target keyframe to the current frame that has the highest spatial similarity to each spatial location in the current frame.

[0093] (2)

[0094] In equation (2), p Indicates target frame i diffusion characteristic map A spatial position on the target frame, q represents the position relative to the target frame. i Diffusion feature map of the nearest target keyframe In this context, D represents a spatial location on the graph, where D is the cosine distance. The smaller the cosine distance, the more similar the semantic features.

[0095] The feature vector of the current frame can be calculated based on the weighted sum of the feature vectors of the previous target keyframe after joint editing and the feature vectors of the subsequent target keyframe after joint editing, so as to realize the propagation of editing effects.

[0096] Equation (3) below shows the calculation of the feature vector of the current frame.

[0097] (3)

[0098] In equation (3), This indicates that after editing and dissemination, it is the [number]th [item / section]. i Frame position p The generated new features Indicates from the current frame i Features extracted from the edited image of the nearest past target keyframe. Indicates from the current frame i Features extracted from the edited image of the nearest future target keyframe. Indicates the first i Frame position p At the corresponding position on the past target keyframes Indicates the first i Frame position p The corresponding position on the future target keyframe. This represents the edited features obtained from the corresponding positions of past target keyframes. This represents the edited features obtained from the corresponding position of the future target keyframe. Represents the weight, where, The value is between 0 and 1.

[0099] Equation (3) allows us to use the feature correspondence established from the original input video to propagate the editing features of the keyframes to every frame of the video. Since the diffusion space of the current frame is based on the inter-frame feature correspondence between adjacent keyframes, the logical consistency of the images between frames can be ensured, thereby ensuring that the features of all frames remain consistent in spatial logic.

[0100] In equation (3), the weights The value can be inversely proportional to the time difference between the target keyframe and the current frame.

[0101] In the example embodiment, for the first [item] that needs to have its editing effects propagated... i Frame (its image content is a cat walking forward in the grass), assuming the previous target keyframe i- With the i The time difference between frames is 0.05 seconds, the next target keyframe i+ The time difference with the i-frame is 0.02 seconds. This is due to the subsequent target keyframe. i+ With the i The smaller the time difference between frames, the greater the weight of its features in the editing propagation. The value is larger. i Each image region in the frame is determined based on inter-frame feature correspondences and with greater reference to the subsequent target keyframe. i+ The images are used to generate corresponding editing effects. For example, the grassy area where the cat walked in each frame is used to generate realistic and natural footprints by referencing the similar positions of adjacent keyframes with specific weights.

[0102] Figure 7 This is a diagram illustrating a video editing process according to an example embodiment of the present disclosure.

[0103] Reference Figure 7 In step S710, a feature extraction operation can be performed. The feature extraction in step S710 can correspond to... Figure 1 To avoid redundancy, step S400 will not be described again here.

[0104] In step S720, the time step t can be set to 1. In step S730, a set of keyframes can be randomly selected from the input video.

[0105] In step S740, the selected set of keyframes can be edited. Step S740 can correspond to... Figure 1 Steps S500 and S600 and Figure 5.

[0106] In step S750, the features of the edited keyframe can be propagated to other frames in the input video. Step S750 may correspond to... Figure 1 Step S700 and Figure 6 .

[0107] In step S760, it can be determined whether the current time step t has reached the denoising time step threshold T. If it is determined that t = T, it indicates that the denoising process has ended, and in step S770, the edited video can be output. If it is determined that t is not equal to T, it indicates that the denoising process has not yet ended. In step S780, the value of time step t is incremented by 1, and then the process proceeds to step S730.

[0108] In the example embodiment, the steps of keyframe editing and editing effect propagation can be repeated until the denoised video is output.

[0109] If the image generation model completes the video editing process over T time steps, to ensure the quality of the video editing, each keyframe should be selected as the target keyframe at least once within those T time steps. If, after T time steps, there are still keyframes that have not been selected as target keyframes, further time steps are needed until all keyframes are selected and directly edited.

[0110] Figure 8 This is a block diagram illustrating a video editing apparatus based on interactive effect propagation according to an example embodiment of the present disclosure.

[0111] Reference Figure 8 The apparatus 800 for editing video according to an example embodiment of the present disclosure may include: a preprocessing module 810, a video perception module 820, a feature extraction module 830, an interactive effect generation module 840, a joint editing module 850, and an editing effect propagation and generation module 860.

[0112] The preprocessing module 810 can receive input video and user prompts, and generate interactively-aware prompts based on the user prompts. In an example embodiment, the preprocessing module 810 can also extract a small number of keyframes from the input video to generate few-sample prompts to guide MLLM.

[0113] The video perception module 820 can identify a set of keyframes based on the input video and interactive perception prompts, and generate scene interaction effect information corresponding to the keyframes. In an example embodiment, the video perception module 820 can be implemented as MLLM to discover more scenes and keyframes.

[0114] In an example embodiment, if no user prompt is received, the video perception module 820 can recommend prompts or interactive effects to the user based on the input video.

[0115] The feature extraction module 830 can extract the diffusion features of each frame in the input video, which can be used to generate inter-frame feature correspondences in subsequent processing.

[0116] The interaction effect generation module 840 can edit keyframes based on scene interaction effect information to generate edited keyframes.

[0117] The joint editing module 850 can jointly edit the edited keyframes to generate jointly edited keyframes.

[0118] The editing effect propagation and generation module 860 can edit other frames of the input video based on the features of the jointly edited keyframes to generate the edited output video.

[0119] In the example embodiment, the feature extraction module 830, the interactive effect generation module 840, the joint editing module 850, and the editing effect propagation and generation module 860 can be implemented as an image generation model based on a diffusion model.

[0120] In the example embodiment, the device 800 may also include other modules. For example, the device 800 may also include a user editing and adjustment module (not shown). The user editing and adjustment module can receive the denoised video output by the editing effect propagation and generation module 860, and based on the user's needs, continue to perform additional editing, or directly output the video.

[0121] The device 800 for editing video according to an example embodiment of the present disclosure may be implemented as a terminal and / or a server.

[0122] The terminal can be an electronic device, such as a mobile phone, tablet computer, multimedia playback device, laptop computer, desktop computer, wearable device, in-vehicle terminal device, virtual reality (VR) device, augmented reality (AR) device, mixed reality (MR) device, etc. The terminal may have a client installed running the target application, which has video editing or video generation functions.

[0123] A server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services. The server can serve as the backend server for the target application, providing backend services to the client of the target application.

[0124] Figures 9 to 11 This is an example illustrating a video editing method based on interactive effect propagation according to an exemplary embodiment of the present disclosure.

[0125] Reference Figure 9 Suppose a user has filmed a video of themselves kicking a soccer ball on a grassy field and wants to add some cool effects to make the video more creative. For example, the user might want to add an effect where "the soccer ball bursts into flames, creating sparks and heat waves around it." To do this, the user could enter the prompt "Kicking a flaming soccer ball, sparks flying everywhere."

[0126] Based on user prompts and input video, interactive prompts can be generated using preset templates and input into the MLLM (Multi-Layered Model). The MLLM generates a three-element array containing scene interaction information, including keyframe numbers, corresponding timelines, and a natural language description of the scene interaction, to guide subsequent image editing. The image generation model can then use the scene interaction information output by the MLLM to generate videos with new interactive effects.

[0127] Reference Figure 10 Suppose a user has filmed a video of their child playing in the living room and is currently modifying video parameters in the video editing interface. The video sensing module detects that the user is in the video editing interface and intelligently recommends an underwater world editing style that matches the scene content, asking the user if they want to apply the editing scheme. The user accepts and issues a command to apply the editing scheme. Furthermore, the user can also request more editing schemes from the video sensing module and preview multiple edited videos.

[0128] The video perception module can generate editing prompts and input them into the MLLM to obtain information on scene interaction effects. The image generation model can generate preview clips of children playing underwater with small fish swimming around them, and the children's movements will produce corresponding dynamic water ripples.

[0129] Reference Figure 11 Suppose a user wants to preview the effects of adding different special effects to a pre-shot rainy scene. The user can input an original video clip of a person walking across a puddle without any physical interaction, and input two different prompts: "The person walks quickly across the puddle, splashing water"; "The person walks quickly across the puddle, splashing water high".

[0130] The image generation model can generate two preview clips with different water splash effects based on two prompts, allowing the user to choose. Furthermore, users can add other interactive elements to the video, such as changing the scene to nighttime or making parked cars move.

[0131] This application discloses a video editing method and apparatus based on interactive effect propagation. It generates video by selecting keyframes for editing and propagating the editing effects, without needing to edit each frame individually. This improves video processing efficiency while ensuring the temporal and logical consistency of the generated video. Furthermore, by jointly editing keyframes and using keyframes before and after the frame to be edited to propagate the editing effects, the editing effects of later frames can be propagated forward. This allows the editing effects to propagate not only backward but also forward, thus overcoming the dependence on single-frame editing effects.

[0132] Furthermore, the video editing method and apparatus based on interactive effect propagation according to the example embodiments of this disclosure can automatically sense addable interactive effects and discover possible interactive scenarios of the input video in an unsupervised manner, greatly improving video processing efficiency and user experience.

[0133] The devices, units, modules, and other components described herein are implemented by hardware components. Examples of hardware components that can be used to perform the operations described herein include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described herein. In other examples, one or more hardware components performing the operations described herein are implemented by computing hardware (e.g., by one or more processors or computers). The processor or computer may be implemented by one or more processing elements, such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field-programmable gate arrays, programmable logic arrays, microprocessors, or any other means or combination of means configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, the processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. The hardware components implemented by the processor or computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described herein. Hardware components can also access, manipulate, process, create, and store data in response to the execution of instructions or software. For simplicity, the singular terms "processor" or "computer" are used in the description of the examples described in this application; however, in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component, or two or more hardware components, may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. Hardware components may have any one or more different processing configurations, examples of which include: a single processor, a standalone processor, a parallel processor, Single Instruction Single Data (SISD) multiple processing, Single Instruction Multiple Data (SIMD) multiple processing, Multiple Instruction Single Data (MISD) multiple processing, and Multiple Instruction Multiple Data (MIMD) multiple processing.

[0134] The methods for performing the operations described in this application are executed by computing hardware (e.g., by one or more processors or a computer), which is implemented to execute instructions or software as described above to perform the operations performed by the methods described in this application. For example, a single operation, or two or more operations, may be executed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be executed by one or more processors, or a processor and a controller, and one or more other operations may be executed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may execute a single operation, or two or more operations.

[0135] Instructions or software for controlling a processor or computer to implement hardware components and perform the methods described above can be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure the processor or computer to operate as a machine or special-purpose computer to perform operations performed by the hardware components and methods described above. In one example, the instructions and / or software include machine code (such as machine code generated by a compiler) that is directly executed by the processor or computer. In another example, the instructions or software include high-level code that is executed by the processor or computer using an interpreter. Those skilled in the art or programmers can readily write the instructions and / or software based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding descriptions in the specification, which disclose algorithms for performing operations performed by the hardware components and methods described above.

[0136] Instructions or software used to control a processor or computer to implement hardware components and perform the methods described above, along with any associated data, data files, and data structures, are recorded, stored, or fixed in, or on, one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage devices, hard disk drive (HDD), solid-state drive (SSD), flash memory, card memory (such as multimedia cards or microcards (e.g., Secure Digital (SD) or Extreme Digital (XD))), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and at least one of any other device, any other device configured to store instructions or software and any associated data, data files and data structures in a non-transitory manner and to provide instructions or software and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the instructions.

[0137] Although various exemplary embodiments have been described, those skilled in the art will understand that various changes in form and detail may be made thereto without departing from the spirit and scope of this disclosure as defined by the claims.

Claims

1. A video editing method based on interactive effects propagation, the video editing method comprising: The system receives an input video and user prompts, where the input video consists of multiple frames and the user prompts are used to guide the modification of visual elements in the input video. Generate interactive prompts based on user-generated prompts; Based on the input video and interactive awareness prompts, a pre-trained multimodal large language model is used to identify a set of key frames and generate scene interaction effect information corresponding to the key frames. Use an image generation model to extract the diffusion features of each frame in the input video; Based on scene interaction effect information, an image generation model is used to edit keyframes to generate edited keyframes. Use an image generation model to jointly edit the edited keyframes to generate jointly edited keyframes; Based on the features of the jointly edited keyframes, an image generation model is used to edit other frames of the input video to generate an edited output video.

2. The video editing method according to claim 1, wherein, The steps for generating interactively-aware prompts based on user-provided prompts include: Generate interactive prompts based on preset templates. Interactive awareness prompts are used to fine-tune the multimodal large language model to guide it in perceiving the scene interaction effects in the input video.

3. The video editing method according to claim 2, wherein, Scene interaction effect information includes: keyframe numbers, timeline information corresponding to the keyframes, and natural language descriptions of scene interactions.

4. The video editing method according to claim 1, in, The image generation model is a neural network model based on the diffusion model. The steps involved in editing keyframes using an image generation model include: At each time step in the generation of the interactive effect, a set of target keyframes is randomly selected from the set of keyframes; Obtain the noisy vector of each target keyframe mapped to the latent space; The image generation model is used to generate edited keyframes based on noisy vectors and scene interaction information.

5. The video editing method according to claim 4, wherein, The steps for co-editing the edited keyframes include: Calculate the self-attention features for each edited keyframe, where the self-attention features include query features, key features, and value features; The key features of the edited keyframes are concatenated to generate extended joint attention; Joint editing of each target keyframe is performed based on extended joint attention. The feature vector of the jointly edited keyframe includes the result of fusing the semantic features of the set of target keyframes.

6. The video editing method according to claim 1, wherein, The steps for extracting the diffusion features of each frame in an input video using an image generation model include: Obtain the noisy vector mapped to the latent space for each frame; Feature vectors are extracted using the self-attention layer of the image generation model to obtain the diffusion features for each frame.

7. The video editing method according to claim 1, wherein, The diffusion features of each frame are used to establish the correspondence between features between frames.

8. The video editing method according to claim 7, wherein, The steps for editing other frames of the input video using an image generation model include: Obtain the nearest previous and next target keyframes to the current frame being edited; Based on diffusion features, establish the inter-frame feature correspondence between the current frame and the previous target keyframe, as well as between the current frame and the next target keyframe; Based on the inter-frame feature correspondence and the feature vectors of the previous and next target keyframes, the feature vector of the current frame is generated. The edited current frame is generated based on the feature vector of the current frame.

9. The video editing method according to claim 8, in, The feature vector of the current frame is calculated based on the weighted sum of the feature vectors of the previous target keyframe and the next target keyframe. The weight value is inversely proportional to the time difference between the target keyframe and the current frame.

10. The video editing method according to claim 1, further comprising: If no prompts are received from the user, prompts are recommended to the user based on the input video.

11. A video editing device based on interactive effects propagation, comprising: The preprocessing module is configured as follows: Receive input video and user prompts, wherein the input video comprises multiple frames and the user prompts indicate textual suggestions that will modify visual elements in the input video; Generate interactive prompts based on user-generated prompts; The video perception module is configured to: identify a set of keyframes based on the input video and interactive perception prompts, and generate scene interaction effect information corresponding to the keyframes; The feature extraction module is configured to extract the diffusion features of each frame in the input video; The interaction effect generation module is configured to: edit keyframes based on scene interaction effect information to generate edited keyframes; The joint editing module is configured to perform joint editing on the edited keyframes to generate jointly edited keyframes; The editing effect propagation and generation module is configured to edit other frames of the input video based on the features of the jointly edited keyframes to generate the edited output video.

12. The video editing apparatus according to claim 11, wherein, The preprocessing module is configured as follows: Generate interactive prompts based on preset templates. Interactive awareness prompts are used to fine-tune the multimodal large language model to guide it in perceiving the scene interaction effects in the input video.

13. The video editing apparatus according to claim 12, wherein, Scene interaction effect information includes: keyframe numbers, timeline information corresponding to the keyframes, and natural language descriptions of scene interactions.

14. The video editing apparatus according to claim 11, wherein, The interactive effects generation module is configured as follows: At each time step in the generation of the interactive effect, a set of target keyframes is randomly selected from the set of keyframes; Obtain the noisy vector of each target keyframe mapped to the latent space; Edited keyframes are generated based on noisy vectors and scene interaction effect information.

15. The video editing apparatus according to claim 14, wherein, The joint editing module is configured as follows: Calculate the self-attention features for each edited keyframe, where the self-attention features include query features, key features, and value features; The key features of the edited keyframes are concatenated to generate extended joint attention; Joint editing of each target keyframe is performed based on extended joint attention. The feature vector of the jointly edited keyframe includes the result of fusing the semantic features of the set of target keyframes.

16. The video editing apparatus according to claim 11, wherein, The feature extraction module is configured as follows: Obtain the noisy vector mapped to the latent space for each frame; Feature vectors are extracted using the self-attention layer of the image generation model to obtain the diffusion features for each frame. The diffusion features of each frame are used to establish the correspondence between features between frames.

17. The video editing apparatus according to claim 16, wherein, The editing effect propagation and generation module is configured as follows: Obtain the nearest previous and next target keyframes to the current frame being edited; Based on diffusion features, establish the inter-frame feature correspondence between the current frame and the previous target keyframe, as well as between the current frame and the next target keyframe; Based on the inter-frame feature correspondence and the feature vectors of the previous and next target keyframes, the feature vector of the current frame is generated. The edited current frame is generated based on the feature vector of the current frame.

18. The video editing apparatus according to claim 17, in, The feature vector of the current frame is calculated based on the weighted sum of the feature vectors of the previous target keyframe and the next target keyframe. The weight value is inversely proportional to the time difference between the target keyframe and the current frame.

19. The video editing apparatus according to claim 11, wherein, The preprocessing module is also configured as follows: If no prompts are received from the user, prompts are recommended to the user based on the input video.

20. A non-transitory computer-readable storage medium storing instructions, which, when executed by a processor, cause the processor to perform the video editing method according to any one of claims 1 to 10.

Images