Global and local frame joint modeling method for zero-sample text-driven video editing

Abstract

The invention discloses a global and local frame joint modeling method for zero-sample text-driven video editing, and belongs to the field of computer vision and video editing. The method is applied to an inverse process of a potential diffusion model for video editing, and the inverse process is that de-noising is carried out on a video frame after noise adding in a potential space by adopting a diffusion model of a U-Net structure, features after text coding are introduced as condition information in the de-noising process, and an edited video frame conforming to text description is generated step by step; the U-Net structure comprises a Transform block, a self-attention block in the Transform block is combined with a global frame feature and a local frame feature for calculation, during calculation of the self-attention block, a key sum value based on global frame modeling and a key sum value based on local frame modeling are generated respectively, global frame modeling and local frame modeling results are merged in a time-space domain feature dimension, and a global frame modeling result and a local frame modeling result are obtained. The original key and the original value are updated, and the self-attention value is calculated using the updated key, the updated value and the original query.

Description

Technical Field

[0001] The present invention belongs to the fields of computer vision and video editing, and particularly relates to a global-local frame joint modeling method for zero-shot text-driven video editing. Background Art

[0002] In recent years, with the rapid development of generative models and natural language processing (NLP) technologies, text-driven video editing technologies have gradually become a research hotspot. Zero-shot methods do not require training or fine-tuning, and their core lies in relying on diffusion models that have been pre-trained with large-scale data. Usually, zero-shot methods use 2D diffusion models because, in the current research and application scenarios, due to their relatively simple structure and extensive application foundation, they are easier to obtain compared to other types of models. And these 2D diffusion models are trained based on image data, which results in their lack of the ability to effectively model video frame sequences.

[0003] Precisely due to the limitations of the above 2D diffusion models, text-driven zero-shot video editing methods mainly focus on solving problems related to temporal coherence and semantic alignment between text and video frames. In the initial stage of the development of video editing methods, the main focus was on the editing effects and feasibility of short videos, and the DDIM inverse process frame by frame was generally used to implement editing operations. Subsequently, in order to improve the temporal coherence of video editing results, researchers introduced cross-frame attention or temporal attention technologies. Although these methods consume relatively less computational power and storage resources and are theoretically more efficient solutions, when actually editing video content, problems such as video incoherence and flickering that affect the visual effect will occur. Therefore, due to these limitations, frame-by-frame processing methods are difficult to be effectively applied in the editing scenarios of long videos. This practical need has promoted the development of multi-frame processing methods, such as expanding the self-attention module used in image processing into an extended attention module to find the internal connections between multiple frames. However, although multi-frame processing methods have improved the deficiencies of frame-by-frame processing to a certain extent, these methods still cannot completely avoid the occurrence of flickering problems. Precisely due to the limitations of the above 2D diffusion models, text-driven zero-shot video editing methods mainly focus on solving problems related to temporal coherence and semantic alignment between text and video frames.

[0004] The STEM method mainly improves existing methods. The direction of improvement lies in discovering that previous improved extended attention schemes only focused on the denoising stage of the DDIM inverse process, while ignoring the crucial issue of the noise addition stage. Moreover, previous methods that merged multiple frames only played a role in the denoising process, and their DDIM inverse process still only considered adjacent frames. This explains why these methods still have the problem of flickering. Additionally, directly applying the DDIM inverse process to multiple frames would lead to unacceptable computational and spatial complexities. The STEM method significantly reduces the computational complexity of this process by applying machine learning techniques to the self-attention in the denoising diffusion model. Not only that, it has successfully solved the video flickering problem and enhanced the coherence of the edited content in the temporal dimension. Although the STEM method has achieved remarkable results in the field of video editing, it is not without limitations. In long video or ultra-high-resolution video editing tasks, the STEM method still faces the problem of insufficient semantic alignment between the text and video frames. Summary of the Invention

[0005] In view of the above problems, the present invention proposes a global-local frame joint modeling method for zero-shot text-driven video editing. The technical solution is as follows:

[0006] A global-local frame joint modeling method for zero-shot text-driven video editing, which is applied to the inverse process of a latent diffusion model for video editing. The inverse process is to denoise the noisy video frames in the latent space using a diffusion model with a U-Net structure, and introduce the features encoded from the text as conditional information during the denoising process to gradually generate the edited video frames that conform to the text description.

[0007] The U-Net structure includes Transformer blocks, and the self-attention blocks in the Transformer blocks jointly calculate the global frame features and local frame features. The specific calculation process of the self-attention blocks is as follows:

[0008] (1) Given the input video frame feature set, obtain the original query, original key, and original value through a linear mapping layer; first use a probabilistic clustering algorithm to compress the original key and value of the set, and then obtain the key and value after global frame modeling by amplifying the compressed tensor.

[0009] (2) Split the input video frame feature set into multiple segments, and each segment respectively obtains the original query, original key, and original value through a linear mapping layer; first use a probabilistic clustering algorithm to compress the original key and value of each segment, and then merge the compressed tensors of each segment in the frame dimension to obtain the key and value after local frame modeling.

[0010] (3) Merge the global frame modeling and local frame modeling results in the spatio-temporal domain feature dimension, update the original keys and original values, and calculate the self-attention values using the updated keys, values, and the original query.

[0011] In addition, a video editing method is also disclosed, which is characterized by including:

[0012] Given the video to be edited and the description text, extract the video frame features and text features using an image encoder and a text encoder respectively;

[0013] Gradually add noise to the video frame features to generate a pure noise image;

[0014] During the reverse denoising process of the diffusion model, introduce the text features as conditional information, and gradually remove the noise in the pure noise image and generate the edited video frame features that conform to the text description;

[0015] Use an image decoder to decode the edited video frame features to generate the edited video;

[0016] The processes of adding noise and denoising are implemented in the latent space using a Unet network alternately composed of ResNet blocks and Transformer blocks as the diffusion model, and during the denoising process, the self-attention blocks in the Transformer blocks calculate the self-attention values using the above-mentioned global-local frame joint modeling method.

[0017] The beneficial effects of the present invention are:

[0018] The present invention proposes a global-local frame joint modeling method for text-driven video editing for zero-shot learning, which improves the disadvantage of poor semantic alignment of existing methods in long video editing by combining global frame visual features and local frame visual features. The present invention can edit longer videos and obtain better editing effects. At the same time, it can also save the GPU memory consumption during program operation. Description of the Drawings

[0019] Figure 1 It is a flowchart for global-local frame joint modeling.

[0020] Figure 2 It is a comparison of GPU memory consumption between GLFM and STEM algorithms.

[0021] Figure 3 It is a comparison of BLIP values per frame when GLFM and STEM edit sample videos.

[0022] Figure 4 It is a comparison of the editing effects of GLFM and STEM for two sample editing tasks. Detailed Embodiments

[0023] The following is a detailed description of the preferred implementation manner of the present invention through selected implementation cases and in combination with the accompanying drawings, aiming to deeply explain the core idea of the present invention, rather than aiming to set any form of limitation. We encourage various flexible innovations and adjustments on the premise of maintaining the basic principles and spirit of the present invention, including but not limited to fine-tuning of the structure, optimization of the function, and expansion of the application scenario, etc. All these variations and innovations are regarded as within the protection scope of the present invention, reflecting the broad inclusion and respect for the boundaries of technological innovation.

[0024] The applicant has summarized through experiments and research that the combination of global frame modeling and local frame modeling is used to solve the poor semantic alignment problem that occurs when the STEM method is used for editing long videos. To solve these problems, the present invention innovatively proposes a global-local frame modeling method. This method improves the semantic alignment between text prompts and video editing content in video editing tasks. In addition, the present invention also has less spatial complexity, that is, it consumes less video card memory during actual operation.

[0025] The present invention is a zero-sample text-driven video editing method, and the main steps are as follows:

[0026] Given the video to be edited and the description text, the video frame features and text features are extracted by using an image encoder and a text encoder respectively; the image encoder uses a variational autoencoder, and the text encoder uses a VIT or BERT encoder;

[0027] Gradually add noise to the video frame features to generate a pure noise image;

[0028] During the reverse denoising process of the diffusion model, the text features are introduced as conditional information, and the noise in the pure noise image is gradually removed to generate the edited video frame features that conform to the text description;

[0029] Use an image decoder to decode the edited video frame features to generate an edited video;

[0030] The processes of adding noise and denoising are implemented in the latent space by using a Unet network alternately composed of ResNet blocks and Transformer blocks as the diffusion model, and during the denoising process, the self-attention block in the Transformer block calculates the self-attention value by using the global-local frame joint modeling method.

[0031] Here, a Unet structure alternately composed of ResNet blocks and Transformer blocks is used as the diffusion model.

[0032] As Figure 1 shown, the global-local frame joint modeling method mainly includes the following steps:

[0033] (1) Input video frame features and set the clustering dimension and the number of video segmentations;

[0034] (2) The video frame features pass through the linear mapping layer of the self-attention module to obtain "queries", "keys", and "values";

[0035] (3) Global frame modeling process, pack the "keys" and "values" corresponding to all video frame features and use the probability clustering algorithm, the EM algorithm, to compress all video frame features;

[0036] (4) Local frame modeling process, divide all video frames into different small segments and use the EM algorithm to compress the "keys" and "values" corresponding to each small segment one by one;

[0037] (5) Amplify the compressed tensors of the "keys" and "values" corresponding to all video frame features in step (3);

[0038] (6) Merge the compressed tensors of the "keys" and "values" corresponding to the small segment video frame features in step (4) in the time domain dimension;

[0039] (7) Merge the small segment tensors and the tensors of all video frames in the spatio-temporal domain dimension;

[0040] (8) Use the compressed tensors to replace the original tensors for self-attention calculation.

[0041] In a specific implementation of the present invention, the specific implementation details of the above steps are as follows.

[0042] (1) Given a video frame feature set X, its size is The clustering dimension k1 for global frame modeling and the clustering dimension k2 for local frame modeling, the number of frames in the video small segment is M = 1; where N represents the number of video frames, C represents the number of channels of the video frame, h represents the number of attention heads in the self-attention block, and n seq represents the number of patch blocks into which the video frame is converted.

[0043] (2) The video frame feature set X passes through the linear mapping layer of the self-attention module to obtain "queries" Q = W Q X, "keys" K = W K X, and "values" V = W V X.

[0044] (3) Global frame modeling process.

[0045] Compress the "values" V and "keys" K of the video frame set X to obtain the compressed tensors and where V and K are obtained by the video frame X passing through the linear mapping layer in the self-attention mechanism.

[0046] In a specific implementation of the present invention, the "values" V and "keys" K of the video frame set X are compressed using the Expectation-Maximization (EM) algorithm to obtain the compressed tensors of all video frames. and The EM algorithm is an iterative algorithm used to estimate the parameters in a probability model.

[0047] Here, under the definition of step (1), the present invention compresses V and K into k1 cluster centers. First, k1 cluster centers are randomly initialized. For each row in V or K, the distance between it and each cluster center (such as the Euclidean distance) is calculated, and the probability of each row belonging to each cluster is calculated based on the distance. The cluster centers are updated according to the probability. This process is looped until the cluster centers converge, and then the original V or K can be replaced by k1 cluster centers. The compression processes of V and K are implemented using the same process.

[0048] (4) Local frame modeling process.

[0049] The video frame set X is segmented into M segment sets and then each segment is compressed one by one of and to obtain a set of compressed tensors of "values" and "keys" and where and

[0050] In a specific implementation of the present invention, the EM algorithm is also used for compression. The difference from step (3) is that the compression object here is each segment of and and under the definition of step (1), the present invention compresses and into k2 cluster centers.

[0051] (5) Amplify and the tensor shapes in the frame dimension to

[0052] In this embodiment, before amplification and the dimensions of the compressed tensors are 1×h×k1×C / h. After copying N copies of the two compressed tensors in the frame dimension and then splicing them respectively, where N is the number of video frames, the dimensions of the amplified compressed tensors are N×h×k1×C / h.

[0053] (6) In the frame dimension, merge the small segment compressions obtained in the local frame modeling process respectively to obtain and

[0054] In this embodiment, before merging and the compressed tensor has a dimension of 1×h×k2×C / h. The compressed tensors of each segment are directly merged in the frame dimension, and the dimension of the merged compressed tensor is N×h×k2×C / h.

[0055] (7) Merging the global frame modeling and the local frame modeling in the spatio-temporal domain feature dimension to obtain and

[0056] In this embodiment, the weighted results of the global frame modeling and the local frame modeling are used as the updated keys and values. The weight of the global frame modeling is defined as α, and the weight of the local frame modeling is defined as β. Then:

[0057]

[0058] (8) Using the compressed tensor to replace the "value" and "key" in the self-attention module of the diffusion model for self-attention calculation:

[0059]

[0060] where Output represents the output self-attention value, Softmax represents the softmax function, Q represents the original query, d represents the vector dimension, respectively represent the updated key and value, the superscript T represents the transpose, and k1 and k2 respectively represent the clustering dimensions of the global frame modeling and the local frame modeling.

[0061] The above global-local frame joint modeling method plays a role in the DDIM inverse process of the diffusion model for editing video content. Comparing the effects of the global-local joint modeling method of the present invention with STEM and DDIM, using the structural similarity (SSIM), peak signal-to-noise ratio (PSNR), BLIP-based text-image semantic correlation score (BLIP score), optical flow error (warping error), and flicker value as evaluation indicators, the results are shown in Table 1.

[0062] Table 1 is the experimental data analysis and comparison of GLFM with the other two methods.

[0063]

[0064] It can be seen from the experiments on videos that the present invention has obvious advantages in the competition of various algorithms.

[0065] Such as Figure 2, where the data marked in black bold indicates that the results of the method are significantly better than those of other methods, "Mean" represents the average index value of each frame, and "std" represents the variance of a certain detection index over all frames. It can be seen that the performance of the present invention is the most balanced among the algorithms used, and it has strong competitiveness in the field of text-driven video editing compared with other algorithms.

[0066] from Figure 3 It can be seen that the consumption of graphics card memory of GLFM proposed by the present invention and the existing method STEM increases with the increase of video frame number. Figure 4 Comparison of editing effects of two editing task examples GLFM and STEM.

[0067] The above examples are only specific embodiments of the present invention. Obviously, the present invention is not limited to the above examples, and many variations are possible. All variations that can be directly derived or associated with the contents disclosed by a person skilled in the art should be considered as the protection scope of the present invention.

Claims

1. A global-local frame joint modeling method for zero-shot text-driven video editing, applied to the inverse process of the latent diffusion model for video editing, characterized by: The inverse process is to denoise the noisy video frames using a diffusion model with a U-Net structure in the latent space, and introduce the text-encoded features as conditional information in the denoising process to gradually generate edited video frames that conform to the text description; The U-Net structure includes a Transformer block, and the self-attention block in the Transformer block performs calculations in conjunction with global frame features and local frame features. The calculation process of the self-attention block is specifically as follows: (1) Given a set of input video frame features, the original query, original key, and original value are obtained through a linear mapping layer; the original key and original value of the set are first compressed using a probabilistic clustering algorithm, and then the compressed tensor is expanded to obtain the key and value based on the global frame modeling; (2) The input video frame feature set is divided into multiple segments, and each segment is subjected to a linear mapping layer to obtain the original query, original key, and original value; the original key and original value of each segment are first compressed using a probabilistic clustering algorithm, and then the compressed tensors of each segment are merged in the frame dimension to obtain the key and value after modeling based on the local frame; (3) Combine the global frame modeling and local frame modeling results in the spatiotemporal feature dimension, update the original key and original value, and use the updated key, value and original query to calculate the self-attention value.

2. The global-local frame joint modeling method for zero-shot text-driven video editing according to claim 1, characterized in that: The dimension of the given input video frame feature set is N×h×n seq ×C / h, the dimension of the compressed tensor before step (1) amplification is 1×h×k1×C / h, and the dimension of the compressed tensor after amplification is N×h×k1×C / h; the dimension of the compressed tensor before step (2) merging is 1×h×k2×C / h, and the dimension of the compressed tensor after merging is N×h×k2×C / h; where N represents the number of video frames, C represents the channel dimension, h represents the number of multi-head attention heads in the self-attention block, k1 and k2 represent the clustering dimensions of global frame modeling and local frame modeling respectively, and n seq Indicates the number of patches that a video frame is converted into.

3. The global-local frame joint modeling method for zero-shot text-driven video editing according to claim 1, characterized in that: The probabilistic clustering algorithm adopts the EM algorithm.

4. The global-local frame joint modeling method for zero-shot text-driven video editing according to claim 1, characterized in that: The weighted results of global frame modeling and local frame modeling are used as updated keys and values.

5. The global-local frame joint modeling method for zero-shot text-driven video editing according to claim 1, characterized in that: The formula for calculating the self-attention value in step (3) is: Among them, Output represents the output self-attention value, Softmax represents the softmax function, Q represents the original query, and d represents the vector dimension. They represent the updated keys and values ​​respectively, the superscript T represents transposition, k1 and k2 represent the clustering dimensions of global frame modeling and local frame modeling respectively.

6. The global-local frame joint modeling method for zero-shot text-driven video editing according to claim 1, characterized in that: The amplified compressed tensor described in step (1) refers to copying the compressed tensor of the original key and the original value after compression in the frame dimension and then splicing them together, where N is the number of video frames.

7. The global-local frame joint modeling method for zero-shot text-driven video editing according to claim 1, characterized in that: A Unet structure consisting of alternating ResNet blocks and Transformer blocks is used as the diffusion model.

8. The global-local frame joint modeling method for zero-shot text-driven video editing according to claim 1, characterized in that: In step (2), each segment contains at least one frame.

9. A video editing method, characterized in that: include: Given a video to be edited and a description text, the image encoder and text encoder are used to extract video frame features and text features respectively; The video frame features are gradually denoised to generate a pure noise image; In the reverse denoising process of the diffusion model, text features are introduced as conditional information to gradually remove the noise in the pure noise image and generate edited video frame features that conform to the text description; Using an image decoder to decode the edited video frame features to generate the edited video; The denoising and denoising processes are implemented in the latent space using a Unet network composed of alternating ResNet blocks and Transformer blocks as a diffusion model, and in the denoising process, the self-attention block in the Transformer block calculates the self-attention value using the global-local frame joint modeling method described in claim 1.

10. The video editing method according to claim 9, characterized in that: The image encoder adopts a variational autoencoder, and the text encoder adopts a VIT or BERT encoder.

Images