Video editing method based on grid layout alternate diffusion and multi-attention control
By segmenting video frames into a grid and applying multi-attention control, the problem of insufficient editing precision and diversity in existing video editing technologies is solved, achieving high-quality video editing effects and improved computational efficiency.
Patent Information
- Application Number
- CN202511001007.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-21
- Publication Date
- 2025-10-24
- Estimated Expiration
- 2045-07-21
AI Technical Summary
Existing video editing technologies are insufficient in terms of editing precision, generation diversity, and semantic consistency, making it difficult to meet the needs of large-scale applications in fields such as film and television production and virtual reality.
By employing a grid layout alternating diffusion and multi-attention control approach, video frames are divided into multiple grids, mapped to a low-dimensional latent space using a variational autoencoder, and embedded with temporal attention and self-attention modules. This is combined with an alternating odd-even time step permutation strategy and dynamic fusion attention graphs to improve editing accuracy and generation diversity.
It improves the fine-grained precision and semantic consistency of video editing, reduces video memory usage, and enhances computing efficiency and resource utilization.
Smart Images
Figure CN120512592B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of video analysis, in particular to a video editing method based on grid layout alternating diffusion and multi-attention control. BACKGROUND
[0002] In recent years, video editing technology based on diffusion model has gradually become mainstream. The core idea is to iteratively denoise in low-dimensional latent space guided by text, generating video content that not only preserves the core information of the source video (such as scene structure and motion trajectory) but also meets the target semantics.
[0003] Existing methods are usually based on pre-trained diffusion models, aligning text prompts with visual features through cross-attention mechanisms, and gradually correcting latent space data to achieve editing. However, existing technologies still have significant defects:
[0004] First, the editing precision is limited, and the model is difficult to accurately control the modification boundary of local regions, such as replacing objects that may cause background distortion or edge blur; second, the generation diversity is insufficient, and the editing results tend to converge under the same text prompt, making it difficult to meet the creative needs of multi-version output. In addition, the long video semantic consistency is poor, and existing methods lack cross-frame temporal constraints, leading to object motion trajectory breaks or morphological mutations.
[0005] The above problems seriously restrict the large-scale application of video editing technology in film and television production, virtual reality, and other fields. SUMMARY
[0006] The purpose of the present application is to provide a video editing method based on grid layout alternating diffusion and multi-attention control to solve the problems raised in the background art, and the core problems to be solved include improving fine-grained editing precision, enhancing generation diversity and semantic consistency, and reducing memory occupancy.
[0007] To achieve the above purpose, the present application provides the following technical solution: a video editing method based on grid layout alternating diffusion and multi-attention control, the method steps of which include:
[0008] S1, dividing the original video frame sequence into multiple grids, each grid containing multiple pixel space video frames arranged continuously, forming grid data;
[0009] S2, mapping the grid data to a low-dimensional latent space through an encoder to generate initial latent space feature data;
[0010] S3, editing the initial latent space feature data, the editing process including a diffusion process and a sampling process, wherein the diffusion process is based on a pre-trained stable diffusion model, a temporal attention module is embedded to enhance the temporal consistency between pixel space video frames, and a self-attention module is used to enhance the local semantic consistency within the grid; in the sampling process, an odd-even time step alternation replacement strategy is performed on the grid layout to promote global consistency across the grid, and the attention maps of the reconstruction branch and the editing branch are dynamically fused according to the timestamp threshold to generate denoised data;
[0011] S4, decoding the denoised data into pixel space video frames by a decoder to obtain edited grid data, and splitting and reorganizing the edited grid data to generate a sequence of edited continuous video frames.
[0012] In the above technical solution, the original video frames are divided into multiple grids, and are mapped to a low-dimensional latent space through a variational autoencoder, which preserves local information within each grid and supports more detailed editing operations; the temporal attention module is embedded in the diffusion model to better capture long-term dependencies between pixel space video frames and improve editing accuracy in the temporal dimension; the self-attention mechanism maintains the spatio-temporal interaction between adjacent pixel space video frames within the grid, ensuring local semantic consistency and improving the quality of fine-grained editing; then in the sampling process, an odd-even time step alternation replacement strategy is applied to promote interaction across the grid by alternating the grid layout between odd and even time stamps, thereby maintaining global consistency and balancing editing freedom and source information protection to ensure the quality of the generated results.
[0013] In the sampling process, the attention maps of the reconstruction branch and the editing branch are dynamically fused according to the timestamp threshold, which specifically includes:
[0014] The preset timestamp threshold, when the time step is greater than the preset timestamp threshold, the cross-attention map, the self-attention map and the time attention map of the reconstruction branch are injected into the editing branch, and the attention maps corresponding to the reconstruction branch and the editing branch are fused through local enhancement;
[0015] When the time step is less than or equal to the timestamp threshold, only the cross-attention map, the self-attention map and the time attention map of the editing branch are retained to respond to the editing instructions of the target text and generate denoised data , which not only ensures the response to the target text instructions, but also improves the diversity of the generated results through local enhancement.
[0016] Further improvement of the technical solution is to adjust the U-Net network in the pre-trained stable diffusion model by embedding a temporal attention module to capture long-term dependencies between pixel space video frames.
[0017] According to the initial latent space feature data The training of the stable diffusion model is performed, the original cross attention module in the stable diffusion model is frozen, and only the weight matrices of the self-attention module and the embedded time attention module are updated, so that the generalization ability of the stable diffusion model to the text instruction is preserved.
[0018] The same objective function as the stable diffusion model is used, and the trainable parameters are iteratively optimized by minimizing the mean square error of the predicted noise and the real noise.
[0019] During the training of the stable diffusion model, the original cross attention module is frozen, and only the weight matrices of the self-attention module and the embedded time attention module are updated, thereby reducing the memory consumption during the training process.
[0020] Further improvement of the technical solution adjusts the dimension of the initial latent space feature data to generate adaptive grid data for adaptive grid segmentation processing.
[0021] The adaptive grid data is split into multiple batches and input to the adjusted U-Net network for noise prediction, wherein the noise prediction includes a self-attention stage, a cross-attention stage, and a time attention stage.
[0022] The self-attention stage in the self-attention module is used to maintain the grid form of the adaptive grid data, perform spatial and temporal interaction of adjacent pixel space video frames within the grid, and ensure local semantic consistency within the grid.
[0023] The cross-attention stage in the cross-attention module is used to split the adaptive grid data into an ordered frame sequence and interact with the conditional information of the target text prompt.
[0024] The time attention stage in the time attention module is used to model the same spatial position pixel points of each pixel space video frame in the time domain, and then restore the grid form.
[0025] The noise prediction results of multiple batches and each stage are merged as the noise prediction result , wherein represents the time step index, represents the conditional information.
[0026] Splitting the adaptive grid data into multiple batches for noise prediction effectively reduces the amount of data processed at a time and reduces the memory burden.
[0027] Further improvement of the technical solution, the inversion formula of the deterministic diffusion implicit model is as follows:
[0028] ,in and Represents the noise scheduling coefficient, which is used to control the weight parameter of the noise intensity; through the inversion formula of the deterministic diffusion implicit model, and after time steps, the noise prediction results Iteratively generate latent space data with controllable noise. The iterative process is arrive ;
[0029] The sampling formula of the deterministic diffusion implicit model is as follows
[0030] ,exist In each time step, the denoised data is generated by step-by-step iteration according to the sampling formula of the deterministic diffusion implicit model and the latent space data with controllable noise. , where the iterative process is arrive ;
[0031] The inversion formula of the deterministic diffusion implicit model and the sampling formula of the deterministic diffusion implicit model support generating multiple versions of results for the same text while ensuring the consistency between frames within each version.
[0032] Compared with the prior art, the present invention has the following beneficial effects:
[0033] Through grid processing, fine-tuning of the stable diffusion model, alternating permutation strategy and dynamic fusion technology, the fine-grained accuracy of video editing is effectively improved, the generation diversity and semantic consistency are enhanced, and the video memory usage is reduced through low-dimensional representation and batch processing. This improves computing efficiency and resource utilization while maintaining high-quality editing effects. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 Schematic diagram of the method steps of the present invention. DETAILED DESCRIPTION
[0035] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.
[0036] See also Figure 1 The present invention provides a technical solution: a video editing method based on grid layout alternating diffusion and multi-attention control, comprising the following method steps:
[0037] S1, divide the original video frame sequence into multiple grids, each grid containing a plurality of pixel space video frames arranged continuously, forming grid data;
[0038] S2, map the grid data to a low-dimensional latent space through an encoder to generate initial latent space feature data;
[0039] S3, editing the initial latent space feature data, the editing process including a diffusion process and a sampling process, wherein the diffusion process is based on a pre-trained stable diffusion model, and a temporal attention embedding module is used to enhance the temporal consistency between pixel space video frames, and a self-attention module is used to enhance the local semantic consistency within the grid; in the sampling process, an odd-even time step alternation replacement strategy is performed on the grid layout to promote global consistency across grids, and an attention map of a reconstruction branch and an editing branch is dynamically fused according to a timestamp threshold to generate denoised data;
[0040] S4, decode the denoised data into pixel space video frames through a decoder to obtain edited grid data, and split and reorganize the edited grid data to generate an edited continuous video frame sequence.
[0041] The specific steps are as follows:
[0042] S1, divide the original video frame sequence into multiple grids, each grid containing a plurality of pixel space video frames arranged continuously, forming grid data;
[0043] Obtain the original video frame sequence and arrange it into multiple grids of a specified dimension (such as 2*2, i.e. one grid stores 4 frames) to obtain grid data;
[0044] Obtain the original video frame sequence, divide the original video frame sequence into multiple grids, each grid containing a plurality of pixel space video frames arranged continuously, forming grid data; wherein the video frame dimension is (8, 3, 512, 512), indicating that the video frame size is 512*512, the channel number is 3, and there are 8 frames in total; divide the original video frame sequence into multiple grid blocks to obtain grid data , each grid contains a fixed number of pixel space video frames, for example, a 2x2 grid contains 4 frames, and the dimension of the grid data becomes (2, 3, 1024, 1024).
[0045] S2, map the grid data to a low-dimensional latent space through an encoder to generate initial latent space feature data , wherein the encoder uses a VAE (variational autoencoder) encoder, and the grid data is mapped to a low-dimensional latent space through the encoder module in the variational autoencoder , wherein represents an encoder module; after encoding using the VAE encoder, an 8-fold downsampling is actually performed, so the data dimension becomes (2, 4, 128, 128).
[0046] S3, editing the initial latent space feature data, the editing process including a diffusion process and a sampling process, wherein the diffusion process is based on a pre-trained stable diffusion model, a temporal attention module is embedded to enhance the temporal consistency between pixel space video frames, and a self-attention module is used to enhance the local semantic consistency within the grid; in the sampling process, an odd-even time step alternation replacement strategy is performed on the grid layout to promote global consistency across the grid, and an attention map of the reconstruction branch and the editing branch is dynamically fused according to a timestamp threshold to generate denoised data, which specifically includes:
[0047] S3.1, adjusting the U-Net network in the pre-trained stable diffusion model (Stable Diffusion), embedding a temporal attention module (Temp-Attention) to capture long-term dependencies between pixel space video frames to enhance the temporal consistency between pixel space video frames; wherein the U-Net network represents a U-shaped convolutional neural network;
[0048] According to the initial latent space feature data training the stable diffusion model, freezing the original cross-attention module (Cross-Attention) in the stable diffusion model, and only updating the weight matrix of the self-attention module (Self-Attention) and the embedded temporal attention module (Temp-Attention), while preserving the generalization ability of the stable diffusion model to text instructions;
[0049] The same objective function as the stable diffusion model is used to minimize the mean square error (MSE) between the predicted noise and the real noise, and the trainable parameters are iteratively optimized.
[0050] S3.2, adjusting the dimension of the initial latent space feature data to generate adaptive grid data ; for example, adjusting to (batch number, channel number, grid frame number, height, width) to adapt to grid segmentation processing;
[0051] Split the adaptive grid data into multiple batches and input to the adjusted U-Net network for noise prediction, wherein noise prediction includes a self-attention phase, a cross-attention phase and a temporal attention phase, wherein:
[0052] The self-attention phase in the self-attention module is used to maintain the grid form of the adaptive grid data, perform spatio-temporal interaction between adjacent pixel space video frames within the grid, and ensure local semantic consistency within the grid.
[0053] The cross-attention stage in the cross-attention module is used to split the adaptive grid data into an ordered frame sequence and interact with the conditional information of the target text prompt;
[0054] The time-attention stage in the time-attention module is used to model the pixels in the same spatial position of each pixel space video frame in the time domain, and restore the grid form after completion;
[0055] Merge the noise prediction results of multiple batches and each stage as the noise prediction result , wherein represents the time step index, represents the conditional information (such as text prompt words, reference images, etc.);
[0056] Through the inversion formula of the deterministic diffusion implicit model (DDIM), and after time steps, the noise prediction result is iteratively generated to obtain the latent space data containing controllable noise, and the inversion formula is as follows:
[0057] , wherein and represent the noise scheduling coefficient, which is a weight parameter for controlling the noise intensity; the iteration process is to .
[0058] S3.3, within time steps, the adjusted U-Net network is used, and the denoising data is iteratively generated according to the sampling formula of the deterministic diffusion implicit model and the latent space data containing controllable noise , wherein the sampling formula of the deterministic diffusion implicit model is as follows
[0059] , wherein the iteration process is to .
[0060] S3.4, the odd-even time step alternation replacement strategy is as follows:
[0061] In the sampling process, within time steps, the layout of the pixel space video frames in the grid is replaced in the odd-even time steps, by arranging all the grids in a row, the even time steps keep the original grid layout unchanged, the odd time steps shift the pixel space video frames in each grid to the right by one frame width along the horizontal direction, the last pixel space video frame is moved to the starting position, and the next even time step restores the original layout, time steps, the grid layout is alternately replaced.
[0062] S3.5, in the denoising sampling of the latent space data containing controllable noise by using the sampling formula, the attention maps of the reconstruction branch and the editing branch are dynamically fused according to the timestamp threshold, as follows:
[0063] When the time step is greater than the preset timestamp threshold, the cross-attention map, the self-attention map and the time-attention map of the reconstruction branch (generated based on the original text) are injected into the editing branch (generated based on the target text), and the attention maps corresponding to the reconstruction branch and the editing branch are fused by replacing, weighting or locally enhancing, so as to ensure that the core information such as the structure and layout of the original video is preserved;
[0064] When the time step is less than or equal to the timestamp threshold, only the cross-attention map, the self-attention map and the time-attention map of the editing branch are retained, so as to fully respond to the editing instructions of the target text; through this phased dynamic fusion mechanism, the source information protection and the target editing freedom are balanced in the denoising data generation process, and finally the denoising data that conforms to the target semantics and maintains the time consistency is generated .
[0065] S4, the denoising data is decoded into pixel space video frames by the decoder to obtain the edited grid data, and the processed grid data is split and reorganized to generate the edited continuous video frame sequence, specifically including:
[0066] The denoising data is decoded into pixel space video frames by the decoder to obtain the edited grid data , The dimension of the denoising data is (2, 3, 2, 1024, 1024); the denoising data is split and reorganized into an ordered frame sequence to generate the edited continuous video frame sequence, at this time, the dimension of the video data is (2, 3, 8, 512, 512), there are two branches of reconstruction and editing, 3 channels, 8 frames of data for each branch, and the height and width are both 512, which are consistent with the original video.
[0067] The basic principles, main features and advantages of the present application are shown and described above. Those skilled in the art should understand that the present application is not limited by the above examples, the above examples and descriptions in the specification are only preferred examples of the present application, and are not intended to limit the present application, various changes and improvements can be made to the present application without departing from the spirit and scope of the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.
Claims
1. A video editing method based on grid layout alternating diffusion and multi-attention control, characterized in that, The method steps are as follows: S1, the original video frame sequence is divided into a plurality of grids, each grid contains a plurality of pixel space video frames arranged continuously, forming grid data; S2, the grid data is mapped to a low-dimensional latent space by an encoder to generate initial latent space feature data; S3, editing the initial latent space feature data, the editing process includes diffusion process and sampling process, wherein the diffusion process is based on a pre-trained stable diffusion model, and a time attention embedding module is used to enhance the time consistency between pixel space video frames, and a self-attention module is used to enhance the local semantic consistency within the grid; In the sampling process, an odd-even time step alternation replacement strategy is performed on the grid layout to promote global consistency across grids, and an attention map of a reconstruction branch and an editing branch is dynamically fused according to a timestamp threshold to generate denoising data; The process of the odd-even time step alternation replacement strategy specifically includes: In the sampling process, in In the sampling process, in In the sampling process, in S4, the denoising data is decoded into pixel space video frames by a decoder to obtain edited grid data, and the edited grid data is split and reorganized to generate an edited continuous video frame sequence.
2. The video editing method based on grid layout alternating diffusion and multi-attention control according to claim 1, wherein, The encoder is a variational autoencoder, and the gridded data is mapped to a low-dimensional latent space by an encoder module in the variational autoencoder to generate initial latent space feature data , , , wherein the encoder module is represented by 3. The video editing method based on grid layout alternating diffusion and multi-attention control according to claim 1, characterized in that: The process of the time attention embedding module specifically includes: Adjust the U-Net network in the pre-trained stable diffusion model, and embed the time attention module to capture the long-time dependence between pixel space video frames; According to the initial latent space feature data The training of the stable diffusion model is performed, the original cross attention module in the stable diffusion model is frozen, only the weight matrices of the self-attention module and the embedded time attention module are updated, and the generalization ability of the stable diffusion model to the text instruction is retained. The same objective function as the stable diffusion model is used to minimize the mean square error of the predicted noise and the real noise to iteratively optimize the trainable parameters.
4. The video editing method based on grid layout alternating diffusion and multi-attention control according to claim 1, wherein, The process of the self-attention module to enhance the local semantic consistency within the grid specifically includes: The dimension of the initial latent space feature data is adjusted to generate adaptive grid data for adaptive grid segmentation processing; Adapting the grid data Splitting into multiple batches, inputting into an adjusted U-Net network for noise prediction, wherein the noise prediction comprises a self-attention phase, a cross-attention phase and a temporal attention phase, wherein: The self-attention stage in the self-attention module is used to maintain the grid shape of the adaptive grid data, perform spatiotemporal interaction between adjacent pixel space video frames within the grid, and ensure local semantic consistency within the grid; The cross-attention stage in the cross-attention module is used to split the adaptive grid data into an ordered frame sequence and interact with the conditional information of the target text prompt; The time attention stage in the time attention module is used to model the pixels at the same spatial position of each pixel space video frame in the time domain, and then restore the grid shape; combining the noise prediction results of the plurality of batches and the individual stages as the noise prediction result wherein denotes a time step index, denotes a condition information.
5. The video editing method based on grid layout alternating diffusion and multi-attention control according to claim 1, wherein, The process of dynamically fusing the attention maps of the reconstruction branch and the editing branch specifically includes: A preset timestamp threshold is set, when the time step is greater than the preset timestamp threshold, the cross-attention map, the self-attention map and the time attention map of the reconstruction branch are injected into the editing branch, and the attention maps corresponding to the reconstruction branch and the editing branch are fused by local enhancement; When the time step is less than or equal to the timestamp threshold, only the cross-attention graph, the self-attention graph and the time-attention graph of the edit branch are retained to respond to the editing instruction of the target text, and the denoising data is generated .
6. The video editing method based on grid layout alternating diffusion and multi-attention control according to claim 5, characterized in that, The denoised data The computational formula includes an inversion formula of a deterministic diffusion implicit model and a sampling formula of the deterministic diffusion implicit model.
7. The video editing method based on grid layout alternating diffusion and multi-attention control according to claim 6, characterized in that, The inversion formula of the deterministic diffusion implicit model is as follows: ， wherein and denotes a noise scheduling coefficient, a weight parameter for controlling the intensity of the added noise; through the inversion formula of the deterministic diffusion implicit model, and through time steps, the noise prediction result is iteratively generated, and the iteration process is to .
8. The video editing method based on grid layout alternating diffusion and multi-attention control according to claim 6, wherein, The sampling formula of the deterministic diffusion implicit model is as follows In At each time step, the denoised data is iteratively generated according to the sampling formula of the deterministic diffusion implicit model and the latent space data with controllable noise Wherein the iteration process is To .
9. The video editing method based on grid layout alternating diffusion and multi-attention control according to claim 1, wherein, The decoding includes: Utilizing a decoder module in a variational autoencoder to decode the denoised data decoded to a pixel space video frame, resulting in edited rasterized data , the split and reorganized into an ordered sequence of frames, generating an edited continuous video frame sequence.