Untrained long video editing method and device based on self-attention and key frame
By employing adaptive attention pruning and keyframe selection, along with self-attention pruning and keyframe selection methods, the problems of high computational resource consumption and content incoherence in long video editing are solved, achieving efficient and high-quality video editing.
Patent Information
- Application Number
- CN202411887053.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-20
- Publication Date
- 2025-11-28
- Estimated Expiration
- 2044-12-20
AI Technical Summary
Existing video editing methods consume a lot of computational resources when processing long videos, making it difficult to maintain the continuity and consistency of video content, especially when the content changes rapidly, resulting in a decline in quality.
An adaptive attention pruning and keyframe selection method is adopted. The key and value sequences in the self-attention are compressed and expanded by adaptive attention pruning, keyframes are dynamically selected, and the inter-frame similarity is calculated based on diffusion features to segment the video. Keyframes are sampled in each video segment, and editing information is propagated using the self-attention matrix.
It significantly reduces computing resource consumption, improves the efficiency and quality of long video editing, and can edit more than a thousand frames of video in a single inference, maintaining the coherence and consistency of video content. It is suitable for video editing in a variety of scenarios and content.
Smart Images
Figure CN119697444B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of video editing technology, and particularly relates to a training-free long video editing method and device based on self-attention and key frames. BACKGROUND
[0002] With the development of deep learning technology, text-driven video editing has become a research hotspot [1, 2, 3]. Existing video editing methods usually require a large amount of computing resources and are difficult to handle long video editing tasks, because they need to process every frame in the video, which leads to a significant increase in computing cost and memory occupation [2, 4, 5, 6]. In addition, in order to maintain the coherence and consistency of the edited video, existing methods usually need to perform extended self-attention mechanism on all video frames, which further increases the computational complexity [1, 4].
[0003] Although there have been some studies trying to solve this problem by simplifying the video editing task into a two-step process of key frame translation and interpolation generation, these methods are still difficult to achieve the editing of minute-level long videos [1, 7, 8]. In addition, these methods usually adopt uniform sampling of key frames, without considering the changes in video content, leading to a decline in video quality in parts with rapid content changes [1].
[0004] REFERENCES
[0005] [1] Geyer, M., Bar-Tal, O., Bagon, S., & Dekel, T. (2023). Tokenflow: Consistent diffusion features for consistent video editing. arXiv preprint arXiv:2307.10373.
[0006] [2] Cong, Y., Xu, M., Simon, C., Chen, S., Ren, J., Xie, Y.,... & He, S. (2023). Flatten: optical flow-guided attention for consistent text-to-video editing. arXiv preprint arXiv:2310.05922.
[0007] [3] Qi, C., Cun, X., Zhang, Y., Lei, C., Wang, X., Shan, Y., & Chen, Q. (2023). Fatezero: Fusing attentions for zero-shot text-based video editing. In Proceedings of the IEEE / CVF International Conference on Computer Vision (pp. 15932-15942).
[0008] [4] Wu, J. Z., Ge, Y., Wang, X., Lei, S. W., Gu, Y., Shi, Y.,... & Shou, M. Z. (2023). Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE / CVF International Conference on Computer Vision (pp. 7623-7633).
[0009] [5] Kara, O., Kurtkaya, B., Yesiltepe, H., Rehg, J. M., & Yanardag, P. (2024). Rave: Randomized noise shuffling for fast and consistent video editing with diffusion models. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 6507-6516).
[0010] [6] Liu, S., Zhang, Y., Li, W., Lin, Z., & Jia, J. (2024). Video-p2p: Video editing with cross-attention control. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 8599-8608).
[0011] [7] Yang, S., Zhou, Y., Liu, Z., & Loy, C. C. (2023, December). Rerender a video: Zero-shot text-guided video-to-video translation. In SIGGRAPH Asia 2023 Conference Papers (pp. 1-11).
[0012] [8] Yang, S., Zhou, Y., Liu, Z., & Loy, C. C. (2024). FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 8703-8712). SUMMARY
[0013] To solve the problems of the prior art, the purpose of the present application is to provide a training-free long video editing method and device based on self-attention and key frames, which realizes text-driven long video editing through adaptive attention pruning and key frame selection, reduces the consumption of computing resources, improves the efficiency and quality of long video editing, and maintains the coherence and consistency of video content.
[0014] The present application adopts the following technical solutions:
[0015] On the one hand, a training-free long video editing method based on self-attention and key frames includes:
[0016] S101, using a pre-trained diffusion model to extract diffusion features of each frame of the original video;
[0017] S102, calculating the similarity between each frame of the original video according to the diffusion features to obtain a similarity heat map between any two frames;
[0018] S103, judging the similarity of the content of each frame based on the similarity heat map, dynamically segmenting the video according to the similarity, and obtaining a plurality of video segments;
[0019] S104, sampling at least one key frame in each video segment;
[0020] S105. Calculate the similarity between the current query key frame and all other key frames based on the diffusion features, obtain a similarity heat map between the current query key frame and all other key frames, and select a preset number of positions with large similarities from the similarity heat map between key frames; obtain a new key matrix and a new value matrix based on the marks corresponding to the preset number of positions; edit the self-attention matrix of the key frames based on the query matrix, the new key matrix, and the new value matrix.
[0021] S106. Calculate the similarity between the key frames and non-key frames in the same video segment according to the diffusion features, obtain the mark that the non-key frame in the same video segment is most similar to the key frame; obtain the corresponding relationship between the marks of the non-key frame and the key frame based on the most similar mark; propagate the edited key frame information to the non-key frame based on the self-attention matrix of the key frame and the corresponding relationship of the marks.
[0022] Preferably, use a pre-trained diffusion model to extract the diffusion features of each frame of the original video, specifically including:
[0023] For the original video with a total of n frames, each frame is represented as I i , use a pre-trained text-to-image diffusion model ∈ θ to extract the corresponding diffusion feature F i ; where, 0 ≤ i ≤ n.
[0024] For the diffusion feature F i corresponding to the i-th frame, the p mark f i p and for the q mark of the feature F j corresponding to the j-th frame First, calculate the cosine similarity between f i p and as follows:
[0025] [[ID=3,5]]
[0026] where, CS represents the calculated cosine similarity; 0 ≤ i < j ≤ n, n represents the number of frames; ||f i p || represents the norm of f i p ; represents the norm of f i p ;
[0027] [[ID=5,2]]Then, calculate the mark q * that is most similar to the mark p:
[0028]
[0029] where T jindicates all labels corresponding to the jth frame;
[0030] Finally, the similarity heat map H is obtained i,j The value of the position corresponding to the label p in the heat map is
[0031]
[0032] Preferably, the similarity of the content of each frame is determined based on the similarity heat map, and the video is dynamically segmented according to the similarity to obtain a plurality of video segments, specifically including:
[0033] The first frame of the video is taken as the start of the first video segment, and the similarity heat maps H 1,2 , H 1,3 , …, H 1,m between the first frame and the subsequent frames are calculated in turn, and when the mean value of the points in the specified region calculated based on the similarity heat map is less than a preset threshold, the first frame to the m-1th frame is taken as the first video segment.
[0034] The mth frame is taken as the start of a new video segment, and the similarity heat maps H m,m+1 , H m,m+2 … between the mth frame and the subsequent frames are calculated in turn until the mean value of the points in the specified region calculated based on the similarity heat map is less than the preset threshold.
[0035] According to the above segmentation method, the entire video is traversed to obtain a plurality of video segments.
[0036] Preferably, the mean value of the points in the specified region is less than the preset threshold, specifically including:
[0037] The mean value of all points in the similarity heat map is less than a first threshold, or the similarity heat map is detected using a sliding window, and when the window exists at a certain position, the mean value of all points in the window is a second threshold.
[0038] Preferably, at least one key frame is sampled in each video segment, specifically including:
[0039] In each denoising step t of the diffusion model, in the video that has been segmented, a frame is randomly selected from each video segment as a key frame.
[0040] Preferably, the self-attention matrix of the key frame is represented as follows:
[0041]
[0042] wherein A k represents the self-attention matrix of the kth key frame; Attention represents the self-attention function; Q k represents the query matrix; denotes a new key matrix; denotes a new value matrix.
[0043] Preferably, in the S106, the key frame is I k , the non-key frame is I o , the spatial position label u of the non-key frame I o corresponding to the most similar spatial position label v of the key frame I k is obtained based on the cosine similarity. * The corresponding relationship between the spatial position label u of the non-key frame I o and the spatial position label v of the key frame I k is represented as follows:
[0044] φ ok (u)=v *
[0045] Based on the self-attention matrix of the key frame and the corresponding relationship of the label, the edited key frame information is propagated to the non-key frame, specifically:
[0046] Each label u in A o corresponding to the oth frame I o is mapped to the corresponding label in the self-attention calculation result A k of the kth frame, which is represented as follows:
[0047] A o [u]=A k [φ ok (u)].
[0048] On the other hand, a training-free long video editing device based on self-attention and key frames includes:
[0049] A diffusion feature extraction module is used to extract the diffusion features of each frame of the original video using a pre-trained diffusion model.
[0050] A similarity heat map calculation module is used to calculate the similarity between each frame of the original video based on the diffusion features, and obtain the similarity heat map between any two frames.
[0051] A video segmentation module is used to determine the similarity of the content of each frame based on the similarity heat map, and dynamically segment the video based on the similarity to obtain multiple video segments.
[0052] A key frame selection module is used to sample at least one key frame in each video segment.
[0053] The self-attention pruning module is used to calculate the similarity between the current query key frame and all other key frames based on the diffusion feature, obtain a similarity heat map between the current query key frame and all other key frames, and select a preset number of positions with high similarity from the similarity heat map between the key frames; based on the labels corresponding to the preset number of positions, a new key matrix and a new value matrix are obtained; and based on the query matrix, the new key matrix and the new value matrix, a self-attention matrix of the key frame is edited.
[0054] The key frame information propagation module is used to calculate the similarity between the key frames and the non-key frames in the same video segment based on the diffusion feature, obtain the most similar labels of the non-key frames and the key frames in the same video segment, obtain the corresponding relationship between the labels of the non-key frames and the key frames based on the most similar labels, and propagate the edited key frame information to the non-key frames based on the self-attention matrix of the key frame and the corresponding relationship of the labels.
[0055] Compared with the prior art, the present application has the following advantages:
[0056] (1) The present application proposes an adaptive attention pruning method for compressing the key and value sequences in the extended self-attention. Specifically, the cosine similarity heat map between the key frames is first calculated using the diffusion feature, then M pixel positions with the highest values in the heat map are selected, and only the labels corresponding to these positions are retained to form new key and value sequences. This pruning process significantly reduces the number of labels involved in the self-attention calculation, thereby reducing the computational complexity while maintaining the coherence and consistency of the video content. The extended self-attention pruning technology of the present application reduces the computational resource consumption in the key frame editing process, significantly improves the efficiency of long video editing, and is particularly suitable for processing video content with a large number of frames.
[0057] (2) The present application also proposes a method for dynamically selecting key frames to improve the efficiency and quality of long video editing. Specifically, the similarity between frames is calculated by analyzing the diffusion feature of the video frames, and the video is dynamically segmented based on these similarities. In each video segment, the method automatically samples key frames that represent the most representative content in the segment. The process of dynamically selecting key frames takes into account the changes in video content, ensuring that more key frames are selected in areas with rapid content changes, and fewer key frames are selected in areas with relatively stable content. This method not only improves the flexibility and controllability of video editing, but also significantly reduces the consumption of computational resources, making long video editing tasks efficient under limited computational resources.
[0058] (3) The present application can realize minute-level long video editing in a single inference, and can edit more than one thousand frames on a single Nvidia A800 graphics processing unit (GPU), which is about ten times longer than existing methods.
[0059] (4) The application can guide the editing process of the video through text prompts, improving the flexibility and controllability of editing;
[0060] (5) The application is not only suitable for specific types of videos, but also can process videos of various scenes and contents, having wide applicability. BRIEF DESCRIPTION OF DRAWINGS
[0061] Figure 1 Flowchart of the self-attention and key frame based training-free long video editing method of the embodiment of the application;
[0062] Figure 2 Example flowchart of the self-attention and key frame based training-free long video editing method of the embodiment of the application;
[0063] Figure 3 Schematic diagram of the key frame selection method of the embodiment of the application;
[0064] Figure 4 Schematic diagram of the adaptive attention pruning method of the embodiment of the application;
[0065] Figure 5 Comparison diagram of the method of the embodiment of the application and other video editing methods;
[0066] Figure 6 Structural block diagram of the self-attention and key frame based training-free long video editing device of the embodiment of the application. DETAILED DESCRIPTION
[0067] The application will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the application and not used to limit the scope of the application. In addition, it should be understood that those skilled in the art can make various modifications or changes to the application after reading the content of the application, and these equivalent forms also fall within the scope of the appended claims of the application.
[0068] In order to more clearly understand the above-mentioned purposes, features and advantages of the application, the application will be described in detail below in conjunction with the drawings and specific embodiments. It should be noted that the embodiments of the application and the features in the embodiments can be combined with each other without conflict.
[0069] In the following description, many specific details are set forth in order to provide a thorough understanding of the application. The described embodiments are merely a part of the embodiments of the application, rather than all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the application.
[0070] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this invention belongs. The terms used in the description of this invention are for the purpose of describing specific embodiments only and are not intended to limit the invention.
[0071] As used in the description and claims of this invention and the above drawings, the term "comprising" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products, or devices.
[0072] See Figure 1 and Figure 2 As shown, a method for editing long videos without training based on self-attention and key frames according to this invention includes the following steps.
[0073] S101. Use a pre-trained diffusion model to extract the diffusion features of each frame of the original video. 4]
[0074] For the original video with a total of n frames, each frame is represented as I[[ID=[]17]] i (0 ≤ i ≤ n), and use the pre-trained text-to-image diffusion model ∈ θ to extract the corresponding diffusion feature F i .
[0075] S102. Calculate the similarity between each frame of the original video based on the diffusion features to obtain a similarity heat map between any two frames.
[0076] ]>Specifically, for the p-marked f i of the diffusion feature F i p corresponding to the i-th frame and the q-marked 4] j of the feature F corresponding to the j-th frame, first calculate the cosine similarity between f i p and as follows: [[ID=[]43]]
[0077] <[]
[0078] where CS represents the calculated cosine similarity; 0 ≤ i < j ≤ n, and n represents the number of frames; ||f<[] i p || represents the norm of f i p 4]> [[ID=[]56]]represents the norm of f i p [[ID=[]59]]; <[]
[0079] Then the token q* most similar to the token p is calculated:
[0080]
[0081] Where T j represents all tokens corresponding to the jth frame;
[0082] Finally, the similarity heat map H i,j is obtained, and the value of the token p in the heat map is :
[0083]
[0084] It should be noted that the token in the present application refers to a high-dimensional (such as 512-dimensional) feature.
[0085] S103, judging the similarity of the content of each frame based on the similarity heat map, dynamically segmenting the video according to the similarity, and obtaining a plurality of video segments.
[0086] Specifically, the first frame of the video is taken as the start of the first video segment, and H 1,2 , H 1,3 , …, H 1,m are calculated in turn until the average value of all points in the calculated heat map H 1,m is less than the first threshold value ms, or the heat map is detected by using a sliding window, and the average value in the window is less than the second threshold value ws when the window is at a certain position, the mth frame is taken as the start of a new video segment, and H m,m+1 , H m,m+2 … are continued to be calculated until the entire video is traversed. Finally, the video segmentation based on the original video is obtained, and the content in each video segment is similar, while the content of different segments is greatly different.
[0087] In the embodiment, ms and ws are two different threshold values, and generally ms < ws. If the average value of all points in the entire heat map is less than ms (indicating that the overall content of the picture has changed), or the average value in a certain sliding window is less than ws (indicating that a certain area of the picture has changed relatively dramatically), the video is divided into a new segment.
[0088] S104, sampling at least one key frame in each video segment.
[0089] Specifically, as shown in Figure 2 and Figure 3 , in each denoising step t of the diffusion model, a frame is randomly selected as a key frame in each segment in the video that has been segmented.
[0090] S105, similarity between the current query key frame and all other key frames is calculated based on the diffusion feature, a similarity heat map between the current query key frame and all other key frames is obtained, and a preset number of positions with high similarity are selected from the similarity heat map between the key frames; based on the labels corresponding to the preset number of positions, a new key matrix and a new value matrix are obtained; and based on the query matrix, the new key matrix and the new value matrix, a self-attention matrix of the key frame is edited and obtained.
[0091] The self-attention matrix of the key frame is represented as follows:
[0092]
[0093] wherein A k represents the self-attention matrix of the k-th key frame; Attention represents the self-attention function; Q k represents the query matrix; represents the new key matrix; represents the new value matrix.
[0094] Referring to FIGS. 1, 2 and 3, Figure 2 and Figure 4 In the embodiment, the computational resource consumption is reduced by retaining the labels in the key and value sequences that are most relevant to the query frame. For each key frame, the following steps are performed: the cosine similarity heat map of this key frame and all other key frames is calculated using the diffusion feature, and the calculation method of the similarity heat map is the same as the implementation in S102. From these heat maps, M positions with the highest values are selected (i.e., for each query, the labels of the positions corresponding to the M highest numerical values are retained to participate in the calculation of attention). Only the labels corresponding to the M positions are retained for the keys and values in the extended self-attention, and new keys and values are obtained, which are much shorter than the default.
[0095] S106, similarity between the key frames and the non-key frames in the same video segment is calculated according to the diffusion feature, and the labels of the key frames that are most similar to the non-key frames in the same video segment are obtained; the correspondence between the labels in the non-key frames and the key frames is obtained based on the most similar labels; and the edited key frame information is propagated to the non-key frames based on the self-attention matrix of the key frames and the correspondence between the labels.
[0096] Given the original video and the obtained video segments, the correspondence between the labels in each non-key frame and the key frames in the same segment is calculated, and the similarity between the key frames and the non-key frames in the same video segment is calculated according to the implementation in S102.
[0097] Let the key frames be I k , the non-key frames be I o , and the non-key frame I oThe spatial position label u of the key frame I k The most similar spatial position label is v * The spatial position label u of the non-key frame I o The spatial position label v of the key frame I k The correspondence between the spatial position label u of the non-key frame I
[0098] φ ok (u)=v *
[0099] For each non-key frame I o , there is a key frame I k in the same video segment, the mapping of each token in the attention matrix A o in the calculation of the extended self-attention of the o-th frame to the corresponding token in the attention matrix A k in the calculation of the extended self-attention of the k-th frame can be expressed as:
[0100] A o [u]=A k [φ ok (u)].
[0101] Further, for each time step t of the diffusion process, the above S104-S106 are repeated until the entire diffusion process is completed, generating a high-quality edited video.
[0102] The method of the present application will be compared with other video editing methods as follows.
[0103] Figure 5 This figure intuitively demonstrates the performance comparison between the long video editing method of the present application and other existing video editing methods. Through a series of carefully designed experiments, this figure demonstrates the effects of different methods in handling the same video editing task, thereby highlighting the superiority of the method of the present application.
[0104] In Figure 5 , the top row shows some frames of the original video, which represent the original state of the video content. The rows below them respectively show the results after processing using different video editing methods. Each row corresponds to a specific editing method, including the method of the present application and several existing video editing techniques.
[0105] Each frame in each row shows different stages of the editing process, or the video content at different time points. Through this layout, Figure 5 the detailed effects of each method in processing video content are clearly demonstrated, including changes in objects, adjustments in background, and conversion of overall style, etc.
[0106] In the application of the method of the present application, the coherence and consistency of the video content are well maintained. Whether it is changing the main object in the video, adjusting the background, or converting the overall style, the method can accurately edit according to the text prompt while ensuring that the parts of the video not mentioned for editing remain consistent with the source video.
[0107] In contrast, other editing methods may have some problems when processing long videos, such as flickering, object distortion, or background inconsistency in some parts of the figure. These problems are usually related to large computational resource consumption and low editing efficiency, while the method of the present application effectively solves these problems through adaptive attention pruning and key frame selection techniques.
[0108] Overall, Figure 5 Not only does it demonstrate the superior performance of the present application in long video editing tasks, but it also highlights its unique advantages in handling complex scenes and maintaining video coherence. These comparison results prove the innovativeness and practicality of the method of the present application in the field of video editing, especially in application scenarios that require processing a large number of frames and maintaining high editing quality.
[0109] Referring to Figure 6 The present application also discloses a training-free long video editing device based on self-attention and key frames, comprising:
[0110] The diffusion feature extraction module 601 is configured to extract diffusion features of each frame of the original video using a pre-trained diffusion model.
[0111] The similarity heat map calculation module 602 is configured to calculate the similarity between each frame of the original video based on the diffusion features to obtain a similarity heat map between any two frames.
[0112] The video segmentation module 603 is configured to determine the similarity of the content of each frame based on the similarity heat map, dynamically segment the video based on the similarity, and obtain a plurality of video segments.
[0113] The key frame selection module 604 is configured to sample at least one key frame in each video segment.
[0114] The self-attention pruning module 605 is configured to calculate the similarity between the current query key frame and all other key frames based on the diffusion features to obtain a similarity heat map between the current query key frame and all other key frames, select a preset number of positions with high similarity from the similarity heat map between the key frames, obtain a new key matrix and a new value matrix based on the labels corresponding to the preset number of positions, and edit a self-attention matrix of the key frame based on the query matrix, the new key matrix, and the new value matrix.
[0115] The key frame information propagation module 606 is configured to calculate the similarity between the key frame and the non-key frame in the same video segment according to the diffusion feature, obtain a most similar label of the non-key frame and the key frame in the same video segment, obtain a corresponding relationship of the labels in the non-key frame and the key frame based on the most similar label, and propagate the edited key frame information to the non-key frame based on the self-attention matrix of the key frame and the corresponding relationship of the labels.
[0116] A specific implementation of a training-free long video editing system based on self-attention and key frames is the same as a training-free long video editing method based on self-attention and key frames. The embodiment will not be repeated.
[0117] It is obvious for those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present application is defined by the appended claims rather than the above description, and all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present application. Any reference signs in the claims should not be regarded as limiting the claims. In addition, it is obvious that the word "comprising" does not exclude other units or, singular does not exclude plural. The plurality of units or systems stated in the system claims can also be implemented by one unit or system through software or hardware.
[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit of the technical solutions of the present application.
Claims
1. A training-free long video editing method based on self-attention and keyframes, characterized in that, include: S101, using a pre-trained diffusion model to extract the diffusion features of each frame of the original video; S102, calculate the similarity between each frame of the original video based on the diffusion characteristics, and obtain a similarity heatmap between any two frames; S103: Based on the similarity heatmap, the similarity of the content of each frame is determined, and the video is dynamically segmented according to the similarity to obtain multiple video segments; S104, sample at least one keyframe in each video segment; S105, calculate the similarity between the current query keyframe and all other keyframes based on diffusion features, obtain the similarity heatmap between the current query keyframe and all other keyframes, and select a preset number of positions with high similarity from the similarity heatmap between keyframes. Based on the tags corresponding to these preset positions, a new key matrix and a new value matrix are obtained; Based on the query matrix, the new key matrix, and the new value matrix, the self-attention matrix of the keyframe is obtained through editing. S106, calculate the similarity between key frames and non-key frames in the same video segment based on the diffusion characteristics, and obtain the most similar label between non-key frames and key frames in the same video segment; obtain the correspondence between the labels in non-key frames and key frames based on the most similar label. Based on the correspondence between the self-attention matrix of keyframes and the labels, the edited keyframe information is propagated to non-keyframes.
2. The training-free long video editing method based on self-attention and keyframes according to claim 1, characterized in that, The pre-trained diffusion model is used to extract the diffusion features of each frame of the original video, specifically including: For the original video with a total of n frames, each frame is represented as I. i Using a pre-trained text-to-image diffusion model ∈ θ Extract the diffusion feature F corresponding to each frame i Where 0≤i≤n.
3. The training-free long video editing method based on self-attention and keyframes according to claim 1, characterized in that, The similarity between frames of the original video is calculated based on diffusion characteristics, resulting in a similarity heatmap between any two frames, specifically including: For the diffusion feature F corresponding to the i-th frame i p-marker f i p and for the feature F of the j-th frame j q tag First, calculate f. i p and The cosine similarity is as follows: Among them, CS represents the calculated cosine similarity; 0 ≤ i < j ≤ n, where n represents the number of frames; ||f i p || represents the norm of f i p ; represents the norm of f i p ; Then, the label q* most similar to label p is calculated: Where T j This represents all tags corresponding to the j-th frame; Finally, the similarity heatmap H is obtained. i,j The value at position p in the heatmap for:
4. The training-free long video editing method based on self-attention and keyframes according to claim 1, characterized in that, The similarity of each frame is determined based on a similarity heatmap. The video is then dynamically segmented according to the similarity to obtain multiple video clips, specifically including: Using the first frame of the video as the starting point of the first video segment, the similarity heatmap H between the first frame and subsequent frames is calculated sequentially. 1,2 H 1,3 ... H 1,m When the mean value of points in a specified region calculated based on the similarity heatmap is less than a preset threshold, the first frame to the (m-1)th frame will be used as the first video segment. Using the m-th frame as the starting point of a new video segment, calculate the similarity heatmap H between the m-th frame and subsequent frames. m,m+1 H m,m+2 ...similar heatmaps between ..., until the mean value of points in a specified region calculated based on the similarity heatmap is less than a preset threshold, then segmentation is performed; By traversing the entire video using the segmentation method described above, multiple video segments can be obtained.
5. The training-free long video editing method based on self-attention and keyframes according to claim 4, characterized in that, The mean value of points in a specified area is less than a preset threshold, specifically including: If the mean of all points in the similarity heatmap is less than the first threshold, or if a sliding window is used to detect the similarity heatmap, and there exists a window at a certain position where the mean of all points within the window is less than the second threshold.
6. The training-free long video editing method based on self-attention and keyframes according to claim 1, characterized in that, Sample at least one keyframe in each video segment, specifically including: In each denoising step t of the diffusion model, a frame is randomly selected from each video segment as a keyframe in the video that has been segmented.
7. The training-free long video editing method based on self-attention and keyframes according to claim 1, characterized in that, The self-attention matrix of the keyframe is represented as follows: Among them, A k The self-attention matrix represents the k-th keyframe; Attention represents the self-attention function; Q represents the self-attention matrix. k Represents the query matrix; Represents the new bond matrix; This represents the new value matrix.
8. The training-free long video editing method based on self-attention and keyframes according to claim 1, characterized in that, In S106, let the keyframe be I. k Non-keyframes are I o Non-keyframe I is obtained based on cosine similarity. o Spatial location marker u corresponds to keyframe I k The most similar spatial location is labeled v * Non-keyframe I o Spatial location marker u and keyframe I k The correspondence of the spatial location marker v* is represented as follows: φ ok (U)=v * Based on the correspondence between the self-attention matrix of keyframes and the labels, the edited keyframe information is propagated to non-keyframes, specifically as follows: The oth frame I o Corresponding A o Each tag u in the graph is mapped to the self-attention calculation result A in the k-th frame. k The corresponding markers in the text are represented as follows: A o [u]=A k [φ ok (u)]。 9. A training-free long video editing device based on self-attention and keyframes, characterized in that, include: The diffusion feature extraction module is used to extract the diffusion features of each frame of the original video using a pre-trained diffusion model. The similarity heatmap calculation module is used to calculate the similarity between each frame of the original video based on the diffusion characteristics, and to obtain a similarity heatmap between any two frames. The video segmentation module is used to determine the similarity of the content of each frame based on the similarity heatmap, and to dynamically segment the video according to the similarity to obtain multiple video clips; The keyframe selection module is used to sample at least one keyframe in each video segment; The self-attention pruning module is used to calculate the similarity between the current query keyframe and all other keyframes based on the diffusion feature, obtain a similarity heatmap between the current query keyframe and all other keyframes, and select a preset number of positions with high similarity from the similarity heatmap between keyframes. Based on the tags corresponding to these preset positions, a new key matrix and a new value matrix are obtained; Based on the query matrix, the new key matrix, and the new value matrix, the self-attention matrix of the keyframe is obtained through editing. The keyframe information propagation module is used to calculate the similarity between keyframes and non-keyframes in the same video segment based on the diffusion characteristics, and to obtain the most similar label between non-keyframes and keyframes in the same video segment; based on the most similar label, the correspondence between the labels in non-keyframes and keyframes is obtained. Based on the correspondence between the self-attention matrix of keyframes and the labels, the edited keyframe information is propagated to non-keyframes.