A zero-shot video editing method based on diffusion model
By introducing ST and SC timing modules into the diffusion model and modifying the cross-attention layer, the problems of high fidelity and timing consistency in video editing are solved, achieving efficient video editing results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2024-01-04
- Publication Date
- 2026-07-10
AI Technical Summary
Existing technologies struggle to achieve high-fidelity text-driven video editing, and require high training resources or suffer from reduced generation quality, making it difficult to guarantee temporal consistency.
A diffusion model is adopted, and the self-attention module is replaced with the ST and SC time-series modules. Combined with the modification of the cross-attention layer, the video is edited using text features and the time-series information of video frames. The edited video is generated using empty text inversion and noise reconstruction techniques.
It achieves high-fidelity video editing, ensuring consistency and timing quality between video frames, and improving the quality and efficiency of video generation.
Smart Images
Figure CN118037569B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision and pattern recognition technology, and in particular relates to a zero-shot video editing method based on a diffusion model. Background Technology
[0002] Image generation models such as DALL·E2, Imagen, and Stable Diffusion have enabled the generation of unprecedentedly diverse and realistic images with complex objects and scenes, ushering in a new era of image generation. As an important application, image editing based on pre-trained diffusion models has also made significant progress. These methods allow users to edit input images with simple text description prompts, achieving high fidelity to the original image while conforming to the target description. Currently, there has been little success in extending this success to the field of video editing: given an input video and text prompts, a text-driven video editing algorithm is required to output an edited video that satisfies the following requirements: (1) text-video alignment: the generated edited video should be aligned with the description of the text prompts; (2) fidelity to the original video: each frame of the edited video should be consistent in content with the corresponding frame of the original video; (3) quality: the generated video should maintain temporal consistency and be of high quality.
[0003] Drawing on the success of text-to-image diffusion models, one approach is to build video editing algorithms on top of pre-trained video diffusion models on large-scale video datasets (Molad E, Horwitz E, Valevski D, et al. Dreamix: Video diffusion models are general video editors[J]. arXiv preprint arXiv:2302.01329,2023.). However, training requires a large amount of paired text-to-video data and computational resources, which are often difficult to obtain. Another approach is to fine-tune pre-trained text-to-image diffusion models based on individual videos for video generation. Although this method is more efficient, each video still requires more than 70M parameters. Furthermore, a small amount of training updates the weights of the pre-trained model, which may lead to a decrease in the quality of the generated content. In addition, although implementing video editing through image frame editing techniques seems easy, even with content-preserving DDIM inversion and cross-attention guidance, frame-by-frame image editing can produce temporally inconsistent results when editing video styles due to the lack of temporal modeling. Summary of the Invention
[0004] To overcome the above problems, the purpose of this application is to provide a zero-sample video editing method based on a diffusion model, which can ensure high fidelity while maintaining temporal quality and video quality.
[0005] According to a first aspect of the embodiments of this application, a zero-shot video editing method based on a diffusion model is provided, comprising:
[0006] Step S1: Obtain the video to be edited with the original description and the target description, and scale each frame of the video to be edited to a predetermined size;
[0007] Step S2: Use a text segmenter to encode the original description and the target description to obtain the corresponding Token ID sequence, then use a text encoder to obtain the original description features and the target description features, and obtain the position sequence of the different tokens in the original description and the target description by comparing the Token IDs.
[0008] Step S3: Encode the scaled video to be edited using an autoencoder in the diffusion model to obtain the initial value z0;
[0009] Step S4: Obtain the corresponding initial noise z from the initial value z0 through inversion using an empty text. T And empty text features;
[0010] Step S5: Replace the forward pass of the self-attention module in the diffusion model with a temporal attention module including the ST temporal module and the SC temporal module, and modify the forward pass of the cross attention layer in the diffusion model so that the cross attention layer can obtain the attention map corresponding to each token calculated by the original text description features and the current image.
[0011] Step S6: Concatenate the empty text features with the original description features and the target text description features according to the Batch dimension, and feed the concatenated features into the diffusion model to generate the video based on the initial noise, and obtain the edited video based on the position sequence.
[0012] Furthermore, in step S4, the guiding coefficient w = 1 during DDIM inversion, and the output is the noise at each time step. in During DDIM sampling, the guiding coefficient is w = 7.5, and the noise at time step T is initialized. For time steps t = T, ..., 1, perform the following optimization:
[0013]
[0014] in This represents the DDIM sampling function. It is an empty text feature encoding, C is the original descriptive feature, and at the end of each step, it is updated.
[0015] Further, in step S5, the forward pass of the self-attention module in the diffusion model is replaced with a temporal attention module including the ST temporal module and the SC temporal module;
[0016] The ST timing module is used to calculate the weights of each feature point of the input feature and all frame feature points as the timing feature;
[0017] The SC timing module is used to calculate the weight of each feature point of the input feature with all feature points of the previous frame and the first frame as the timing feature.
[0018] Furthermore, in the ST timing module:
[0019] For the feature hs input to the time-series module, its feature dimension is [(BF)ND], where B is the batch size, F is the number of frames, N is the number of features, and D is the length of each feature. This feature is first processed by the Q, K, and V weight matrices in the time-series module to obtain the corresponding query, key, and value, with the same feature dimension as the input, both having a dimension of 3. For the key and value, the second dimension is expanded to obtain [B 1F ND], and then the second dimension is copied F times to obtain [BFFND]. Finally, the original dimension number is restored, i.e., [BFFND] -> [(BF)(FN)D]. After completing the above operations, the dimension for the query is [(BF)ND], and the dimension for the key and value is [(BF)(FN)D]. The attention formula is then used to calculate the value according to the current state.
[0020]
[0021] Thus, QK is obtained. T The dimension is [(BF)N(FN)], which represents the attention map of each frame feature point and all frame feature points.
[0022] Furthermore, in the SC timing module, features related to the previous frame and the first frame are calculated:
[0023] Get the previous frame number of each frame number The key and value in the temporal self-attention layer are then truncated according to their sequence numbers and concatenated according to the third dimension, i.e.:
[0024]
[0025] Furthermore, step S6 specifically includes:
[0026] The empty text features and the original description features are concatenated according to the Batch dimension to obtain the original concatenated features;
[0027] The empty text features and the target text description features are concatenated according to the Batch dimension to obtain the target concatenated features;
[0028] The original splicing features and initial noise are fed into a diffusion model to obtain the video to be edited;
[0029] The target splicing features and initial noise are fed into a diffusion model to obtain a preliminary edited video;
[0030] Based on the position sequence obtained in step S2, the attention map of the preliminary edited video at the corresponding position is replaced with the attention map of the video to be edited at the corresponding position, thereby generating the edited video.
[0031] According to a second aspect of the embodiments of this application, a zero-shot video editing apparatus based on a diffusion model is provided, comprising:
[0032] Acquisition module: Acquires the video to be edited with the original description and the target description, and scales each frame of the video to be edited to a predetermined size;
[0033] Text encoding module: The original description and target description are encoded by a text segmenter to obtain the corresponding Token ID sequence. Then, the original description features and target description features are obtained by a text encoder. By comparing the Token IDs, the position sequence of the tokens that are different in the original description and target description is obtained.
[0034] Video encoding module: Encodes the scaled video to be edited using an autoencoder in the diffusion model to obtain the initial value z0;
[0035] Inversion module: Inverts the initial value z0 using an empty text to obtain its corresponding initial noise z. T And empty text features;
[0036] Model building module: Replace the forward pass of the self attention module in the diffusion model with a temporal attention module including the ST temporal module and the SC temporal module, and modify the forward pass of the cross attention layer in the diffusion model so that the cross attention layer can obtain the attention map corresponding to each token calculated by the original text description features and the current image.
[0037] Video editing module: The empty text features are concatenated with the original description features and the target text description features according to the Batch dimension. The concatenated features are fed into the diffusion model to generate the video based on the initial noise, and the edited video is obtained based on the position sequence.
[0038] According to a third aspect of the embodiments of this application, an electronic device is provided, comprising:
[0039] One or more processors;
[0040] Memory, used to store one or more programs;
[0041] When the one or more programs are executed by the one or more processors, the one or more processors perform the method as described in the first aspect.
[0042] According to a third aspect of the embodiments of this application, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method as described in the first aspect.
[0043] The technical solutions provided by the embodiments of this application may include the following beneficial effects:
[0044] As can be seen from the above embodiments, this application proposes a video inversion method for video editing, which can solve the problem of fidelity between the edited video and the original video and ensure consistency between video frames. This invention proposes a temporal modeling method that can effectively utilize the temporal information of the original video and perform bidirectional temporal modeling, thereby ensuring the temporal quality of the generated video. This invention proposes a spatial regularization method, in which the cross-attention map generated during the reconstruction of inversion noise contains the spatial information of the original video. For this reason, we can use the cross-attention map as spatial regularization and force the model to focus on and be aware of relevant regions through attention map injection.
[0045] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description
[0046] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0047] Figure 1 This is a flowchart illustrating a zero-sample video editing method based on a diffusion model, according to an exemplary embodiment.
[0048] Figure 2 This is a schematic diagram illustrating video editing in a zero-sample video editing method based on a diffusion model, according to an exemplary embodiment.
[0049] Figure 3 This is a block diagram illustrating a zero-sample video editing apparatus based on a diffusion model, according to an exemplary embodiment.
[0050] Figure 4 This is a schematic diagram of an electronic device according to an exemplary embodiment. Detailed Implementation
[0051] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.
[0052] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.
[0053] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."
[0054] Figure 1 This is a flowchart illustrating a zero-sample video editing method based on a diffusion model according to an exemplary embodiment, such as... Figure 1 As shown, this method, when applied to a terminal, may include the following steps:
[0055] Step S1: Obtain the video to be edited with the original description and the target description, and scale each frame of the video to be edited to a predetermined size;
[0056] Specifically, OpenCV is used to read the video, split it into frames, and then each frame is scaled to a specified size (such as 512x512 resolution) before being reassembled into a video.
[0057] In addition, for video editing, it is necessary to first provide a text description of the input video (original text description) and a text description of the desired modification of the video (target text description) so that the model can understand what needs to be modified.
[0058] Step S2: Use a text segmenter to encode the original description and the target description to obtain the corresponding Token ID sequence, then use a text encoder to obtain the original description features and the target description features, and obtain the position sequence of the different tokens in the original description and the target description by comparing the Token IDs.
[0059] Specifically, StableDiffusion v1.5 is selected as the base model. The text segmenter in it is used to segment the original text description and the target text description, and the resulting sequences are compared. The sequence number of the difference is taken.
[0060] Step S3: Encode the scaled video to be edited using the autoencoder in the diffusion model DDIM to obtain the initial value;
[0061] Specifically, the video pixel values are normalized to between 0 and 1 and then fed into the autoencoder to obtain the initial value z0.
[0062] Step S4: Obtain the corresponding initial noise z from the initial value z0 through inversion using an empty text. T And empty text features;
[0063] Specifically, during DDIM inversion, the guiding coefficient w = 1, and the output is the noise at each time step. in During DDIM sampling, the guiding coefficient is w = 7.5, and the noise at time step T is initialized. For time steps t = T, ..., 1, perform the following optimization:
[0064]
[0065] in This represents the DDIM sampling function. It is an empty text feature encoding, where C is the input text description feature, which here refers to the feature of the original description after being encoded by the text encoder. At the end of each step, it is updated.
[0066] This optimization process allows what has been learned to be more effective. It can reconstruct the original video together with the input C.
[0067] Step S5: Replace the forward pass of the self-attention module in the diffusion model with a temporal attention module including the ST temporal module and the SC temporal module, and modify the forward pass of the cross attention layer in the diffusion model so that the cross attention layer can obtain the attention map corresponding to each token calculated by the original text description features and the current image.
[0068] Specifically, only the forward process of the original self-attention layer in the diffusion model is modified, while the other parameters remain the same as the original self-attention layer.
[0069] For the spatial-temporal (ST) temporal module, the goal is to calculate the weights of each feature point of the input feature with all frame feature points as temporal features. Specifically, the input feature `hs` has a feature dimension of [(BF)ND], where B is the batch size, F is the number of frames, N is the number of features, and D is the length of each feature. First, the Q, K, and V weight matrices in the temporal module are used to obtain the corresponding query, key, and value, with the same feature dimensions as the input (3 for each). For the key and value, the second dimension is expanded to [B 1 FND], then the second dimension is copied F times to obtain [BFFND]. Finally, the original number of dimensions is restored: [BFF ND] -> [(BF)(FN)D]. After these operations, the query dimension is [(BF)ND], and the key and value dimensions are [(BF)(FN)D]. The attention formula is then used to calculate the value according to the current state.
[0070]
[0071] Based on this formula, we can see that QK T The dimension is [(BF)N(FN)], which represents the attention map between the feature points of each frame and the feature points of all frames. Therefore, the result after the attention formula can be used as a temporal feature.
[0072] For the SC (Sparse-Causal) time series module, the goal is to calculate the weights of each feature point in the input features with all feature points in the previous and first frames as the time series features. The specific approach is the same as the ST time series module, except that in the SC time series module, only the features with the previous and first frames need to be calculated. First, obtain the sequence number of the previous frame for each frame. The key and value in the temporal self-attention layer are then truncated according to their sequence numbers, and finally concatenated according to the third dimension, i.e.:
[0073]
[0074] The forward pass of the cross-attention layer in the diffusion model is modified so that the cross-attention layer can obtain the attention map corresponding to each token calculated from the original text description features and the current image. That is, during the forward pass, QK in the cross-attention module is saved. T The calculated result.
[0075] Step S6: Concatenate the empty text features with the original description features and the target text description features according to the Batch dimension, and feed the concatenated features into the diffusion model to generate the video based on the initial noise, and obtain the edited video based on the position sequence;
[0076] Specifically, such as Figure 2 As shown, in the attention map obtained in step S8, select all attention maps corresponding to the position sequence numbers obtained in step S2 in the first batch (the video to be edited reconstructed based on the original descriptive features, i.e., the original video), and replace them with the corresponding positions in the second batch (the preliminary edited video generated based on the target descriptive features), thereby achieving the purpose of editing.
[0077] If we don't do it this way, we need to run the model in two stages. In the first stage, we feed the original text description and corresponding initial noise into the model for generation, and save the intermediate results of the attention feature maps during the generation process. In the second stage, we feed the target text description features and corresponding initial noise into the model for generation, and during the generation process, we replace the attention feature maps with the corresponding feature maps saved in the first stage according to their numbers.
[0078] After the complete diffusion model denoising process, the second batch is taken as the editing result, and the edited video is obtained.
[0079] Corresponding to the aforementioned embodiments of the zero-shot video editing method based on the diffusion model, this application also provides embodiments of a zero-shot video editing apparatus based on the diffusion model.
[0080] Figure 3 This is a block diagram of a zero-sample video editing apparatus based on a diffusion model, according to an exemplary embodiment. (Refer to...) Figure 3 The device may include:
[0081] Acquisition module: Acquires the video to be edited with the original description and the target description, and scales each frame of the video to be edited to a predetermined size;
[0082] Text encoding module: The original description and target description are encoded by a text segmenter to obtain the corresponding Token ID sequence. Then, the original description features and target description features are obtained by a text encoder. By comparing the Token IDs, the position sequence of the tokens that are different in the original description and target description is obtained.
[0083] Video encoding module: Encodes the scaled video to be edited using an autoencoder in the diffusion model to obtain the initial value z0;
[0084] Inversion module: Inverts the initial value z0 using an empty text to obtain its corresponding initial noise z. T And empty text features;
[0085] Model building module: Replace the forward pass of the self attention module in the diffusion model with a temporal attention module including the ST temporal module and the SC temporal module, and modify the forward pass of the cross attention layer in the diffusion model so that the cross attention layer can obtain the attention map corresponding to each token calculated by the original text description features and the current image.
[0086] Video editing module: The empty text features are concatenated with the original description features and the target text description features according to the Batch dimension. The concatenated features are fed into the diffusion model to generate the video based on the initial noise, and the edited video is obtained based on the position sequence.
[0087] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.
[0088] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0089] Accordingly, this application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the diffusion-based zero-sample video editing method described above. Figure 4 The diagram shown is a hardware structure diagram of any device with data processing capabilities, including a zero-sample video editing device based on a diffusion model provided in an embodiment of the present invention. (Except for...) Figure 4 In addition to the processor, memory, and network interface shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.
[0090] Accordingly, this application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the zero-sample video editing method based on the diffusion model described above. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.
[0091] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.
[0092] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.
Claims
1. A zero-shot video editing method based on a diffusion model, characterized in that, include: Step S1: Obtain the video to be edited with the original description and the target description, and scale each frame of the video to be edited to a predetermined size; Step S2: Use a text segmenter to encode the original description and target description to obtain the corresponding Token ID sequence, then use a text encoder to obtain the original description features and target description features, and obtain the position sequence of the different tokens in the original description and target description by comparing the Token IDs. Step S3: Encode the scaled video to be edited using an autoencoder in the diffusion model to obtain the initial value z0; Step S4: Obtain the corresponding initial noise z from the initial value z0 through inversion using an empty text. T And empty text features; Step S5: Replace the forward pass of the self-attention module in the diffusion model with a temporal attention module including the ST temporal module and the SC temporal module, and modify the forward pass of the cross attention layer in the diffusion model so that the cross attention layer can obtain the attention map corresponding to each token calculated by the original text description features and the current image. Step S6: Concatenate the empty text features with the original description features and the target text description features according to the Batch dimension, and feed the concatenated features into the diffusion model to generate the video based on the initial noise, and obtain the edited video based on the position sequence.
2. The method according to claim 1, characterized in that, In step S4, the guiding coefficient w = 1 during DDIM inversion, and the output is the noise at each time step. in During DDIM sampling, the guiding coefficient is w = 7.5, and the noise at time step T is initialized. For time steps t = T, ..., 1, perform the following optimization: in This represents the DDIM sampling function. It is an empty text feature encoding, C is the original descriptive feature, and at the end of each step, it is updated.
3. The method according to claim 1, characterized in that, In step S5, the forward pass of the self-attention module in the diffusion model is replaced with a temporal attention module that includes the ST temporal module and the SC temporal module. The ST timing module is used to calculate the weights of each feature point of the input feature and all frame feature points as the timing feature; The SC timing module is used to calculate the weight of each feature point of the input feature with all feature points of the previous frame and the first frame as the timing feature.
4. The method according to claim 1, characterized in that, In the ST timing module: For the feature hs input to the time-series module, its feature dimension is [(BF)ND], where B is the batch size, F is the number of frames, N is the number of features, and D is the length of each feature. This feature is first processed by the Q, K, and V weight matrices in the time-series module to obtain the corresponding query, key, and value, with the same feature dimension as the input, each having 3 dimensions. For the key and value, the second dimension is expanded to obtain [B 1 FND], and then the second dimension is copied F times to obtain [BFFND]. Finally, the original number of dimensions is restored, i.e., [BFFND] -> [(BF)(FN)D]. After completing the above operations, the dimension for the query is [(BF)ND], and the dimension for the key and value is [(BF)(FN)D]. The attention formula is then used to calculate the value according to the current state. Thus, QK is obtained. T The dimension is [(BF)N(FN)], which represents the attention map of each frame feature point and all frame feature points.
5. The method according to claim 1, characterized in that, In the SC timing module, features related to the previous frame and the first frame are calculated: Get the previous frame number of each frame number The key and value in the temporal self-attention layer are then truncated according to their sequence numbers and concatenated according to the third dimension, i.e.:
6. The method according to claim 1, characterized in that, Step S6 is as follows: The empty text features and the original description features are concatenated according to the Batch dimension to obtain the original concatenated features; The empty text features and the target text description features are concatenated according to the Batch dimension to obtain the target concatenated features; The original splicing features and initial noise are fed into a diffusion model to obtain the video to be edited; The target splicing features and initial noise are fed into a diffusion model to obtain a preliminary edited video; Based on the position sequence obtained in step S2, the attention map of the preliminary edited video at the corresponding position is replaced with the attention map of the video to be edited at the corresponding position, thereby generating the edited video.
7. A zero-shot video editing device based on a diffusion model, characterized in that, include: Acquisition module: Acquires the video to be edited with the original description and the target description, and scales each frame of the video to be edited to a predetermined size; Text encoding module: The original description and target description are encoded by a text segmenter to obtain the corresponding Token ID sequence. Then, the original description features and target description features are obtained by a text encoder. By comparing the Token IDs, the position sequence of the tokens that are different in the original description and target description is obtained. Video encoding module: Encodes the scaled video to be edited using an autoencoder in the diffusion model to obtain the initial value z0; Inversion module: Inverts the initial value z0 using an empty text to obtain its corresponding initial noise z. T And empty text features; Model building module: Replace the forward pass of the self attention module in the diffusion model with a temporal attention module including the ST temporal module and the SC temporal module, and modify the forward pass of the cross attention layer in the diffusion model so that the cross attention layer can obtain the attention map corresponding to each token calculated by the original text description features and the current image. Video editing module: The empty text features are concatenated with the original description features and the target text description features according to the Batch dimension. The concatenated features are fed into the diffusion model to generate the video based on the initial noise, and the edited video is obtained based on the position sequence.
8. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.
9. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, this instruction implements the steps of the method as described in any one of claims 1-6.