Training-free long video editing method and device based on self-attention and key frame

Abstract

The invention discloses a training-free long video editing method and device based on self-attention and key frames, and relates to the technical field of video editing, and the method comprises the steps: employing a pre-trained diffusion model to extract the diffusion feature of each frame of an original video; calculating the similarity between the frames of the original video according to the diffusion characteristics to obtain a similarity heat map between any two frames; judging the similarity of each frame content based on the similarity heat map, and dynamically segmenting the video according to the similarity to obtain a plurality of video clips; sampling at least one key frame in each video clip; calculating the similarity between the current query key frame and all other key frames based on diffusion features, and performing adaptive attention pruning on the key frames; and spreading the edited key frame information to other frames in the video. Through an adaptive attention mechanism and a key frame selection technology, video content continuity and consistency can be maintained, computing resource consumption is reduced, and video editing quality and efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video editing technology, and in particular to a training-free long video editing method and device based on self-attention and key frames. Background Art

[0002] With the development of deep learning technology, text-driven video editing has become a research hotspot [1,2,3]. Existing video editing methods usually require a lot of computing resources and have difficulty handling long video editing tasks because they need to process every frame in the video, which leads to a significant increase in computational cost and memory usage [2,4,5,6]. In addition, in order to maintain the coherence and consistency of the edited video, existing methods usually need to perform an extended self-attention mechanism on all video frames, which further increases the computational complexity [1,4].

[0003] Although some studies have attempted to address this problem by simplifying the video editing task into a two-step process of keyframe translation and interpolation generation, these methods still have difficulty editing minute-long videos [1,7,8]. In addition, these methods usually use uniformly sampled keyframes without considering the changes in video content, resulting in a decrease in video quality in parts where the content changes rapidly [1].

[0004] References:

[0005] [1]Geyer, M., Bar-Tal, O., Bagon, S., & Dekel, T. (2023). Tokenflow: Consistentdiffusion features for consistent video editing. arXiv preprint arXiv:2307.10373.

[0006] [2]Cong,Y.,Xu,M.,Simon,C.,Chen,S.,Ren,J.,Xie,Y.,...&He,S.(2023).Flatten: optical flow-guided attention for consistent text-to-videoediting.arXiv preprint arXiv:2310.05922.

[0007] [3]Qi,C.,Cun,X.,Zhang,Y.,Lei,C.,Wang,X.,Shan,Y.,&Chen,Q.(2023).Fatezero:Fusing attentions for zero-shot text-based video editing.InProceedings of the IEEE / CVF International Conference on Computer Vision(pp.15932-15942).

[0008] [4]Wu,J.Z.,Ge,Y.,Wang,X.,Lei,S.W.,Gu,Y.,Shi,Y.,...&Shou,M.Z.(2023).Tune-a-video:One-shot tuning of image diffusion models for text-to-videogeneration.In Proceedings of the IEEE / CVF International Conference onComputer Vision(pp.7623-7633).

[0009] [5]Kara,O.,Kurtkaya,B.,Yesiltepe,H.,Rehg,J.M.,&Yanardag,P.(2024).Rave:Randomized noise shuffling for fast and consistent video editing withdiffusion models.In Proceedings of the IEEE / CVF Conference on Computer Visionand Pattern Recognition(pp.6507-6516).

[0010] [6]Liu,S.,Zhang,Y.,Li,W.,Lin,Z.,&Jia,J.(2024).Video-p2p:Video editingwith cross-attention control.In Proceedings of the IEEE / CVF Conference onComputer Vision and Pattern Recognition(pp.8599-8608).

[0011] [7] Yang, S., Zhou, Y., Liu, Z., & Loy, CC (2023, December). Rerender a video: Zero-shot text-guided video-to-video translation. In SIGGRAPH Asia2023Conference Papers (pp.1-11).

[0012] [8] Yang, S., Zhou, Y., Liu, Z., & Loy, CC (2024). FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation. In Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition (pp.8703-8712). Summary of the invention

[0013] In view of the problems of the prior art, the purpose of the present invention is to provide a training-free long video editing method and device based on self-attention and key frames, which realizes text-driven long video editing through adaptive attention pruning and key frame selection, improves the efficiency and quality of long video editing by reducing computing resource consumption, and maintains the coherence and consistency of video content.

[0014] The present invention adopts the following technical solution:

[0015] On the one hand, a training-free long video editing method based on self-attention and keyframes, comprising:

[0016] S101, using the pre-trained diffusion model to extract the diffusion features of each frame of the original video;

[0017] S102, calculating the similarity between frames of the original video according to the diffusion feature, and obtaining a similarity heat map between any two frames;

[0018] S103, determining the similarity of the contents of each frame based on the similarity heat map, and dynamically segmenting the video according to the similarity to obtain multiple video segments;

[0019] S104, sampling at least one key frame in each video segment;

[0020] S105. Calculate the similarity between the current query key frame and all other key frames based on the diffusion features, obtain the similarity heat map between the current query key frame and all other key frames, and select a preset number of positions with large similarities from the similarity heat map between key frames; based on the marks corresponding to the preset number of positions, obtain a new key matrix and a new value matrix; based on the query matrix, the new key matrix, and the new value matrix, edit to obtain the self-attention matrix of the key frames.

[0021] S106. Calculate the similarity between the key frames and non-key frames in the same video segment according to the diffusion features, obtain the mark that the non-key frame in the same video segment is most similar to the key frame; based on the most similar mark, obtain the corresponding relationship between the marks of the non-key frame and the key frame; based on the self-attention matrix of the key frames and the corresponding relationship of the marks, spread the edited key frame information to the non-key frames.

[0022] Preferably, use a pre-trained diffusion model to extract the diffusion features of each frame of the original video, specifically including:

[0023] For the original video with a total of n frames, each frame is represented as I i , use the pre-trained text-to-image diffusion model ∈ θ to extract the corresponding diffusion feature F i for each frame; where, 0 ≤ i ≤ n.

[0024] For the p mark f i corresponding to the i-th frame's diffusion feature F i p and for the q mark j of the j-th frame's feature F First, calculate the cosine similarity between f i p and as follows:

[0025]

[0026] where, CS represents the calculated cosine similarity; 0 ≤ i < j ≤ n, n represents the number of frames; ||f i p || represents the norm of f i p ; represents the norm of f i p ;

[0027] Then, calculate the mark q * that is most similar to the mark p:

[0028]

[0029] where T jRepresents all the tags corresponding to the jth frame;

[0030] Finally, we get the similarity heat map H i,j , the value of the position corresponding to the mark p in the heat map for:

[0031]

[0032] Preferably, the similarity of the content of each frame is determined based on the similarity heat map, and the video is dynamically segmented according to the similarity to obtain multiple video segments, which specifically includes:

[0033] Take the first frame of the video as the beginning of the first video segment, and calculate the similarity heat map H between the first frame and the subsequent frames in sequence 1,2 , H 1,3 ,……,H 1,m , when the mean value of the points in the designated area calculated based on the similarity heat map is less than a preset threshold, the first frame to the m-1th frame are taken as the first video segment;

[0034] Take the mth frame as the start of a new video clip and calculate the similarity heat map H between the mth frame and subsequent frames in sequence m,m+1 , H m,m+2 ... until the mean value of the points in the specified area calculated based on the similarity heat map is less than a preset threshold value, segmentation is performed;

[0035] The entire video is traversed according to the above segmentation method to obtain multiple video clips.

[0036] Preferably, the mean value of the points in the specified area is less than a preset threshold, specifically including:

[0037] The mean of all points in the similarity heat map is less than the first threshold, or, when the similarity heat map is detected using a sliding window, when there is a window at a certain position, the mean of all points in the window is the second threshold.

[0038] Preferably, sampling at least one key frame in each video clip specifically includes:

[0039] At each denoising step t of the diffusion model, a frame is randomly selected from each video segment in the segmented video as a key frame.

[0040] Preferably, the self-attention matrix of the key frame is expressed as follows:

[0041]

[0042] Among them, A k represents the self-attention matrix of the k-th key frame; Attention represents the self-attention function; Q k represents the query matrix; represents the new bond matrix; Represents the new value matrix.

[0043] Preferably, in S106, let the key frame be I k , non-key frame is I o , based on cosine similarity, obtain non-key frame I o The spatial position mark u corresponds to the key frame I k The most similar spatial position is marked as v * , non-keyframe I o The spatial position mark u and key frame I k The corresponding relationship of the spatial position mark v* is expressed as follows:

[0044] φ ok (u) = v *

[0045] Based on the correspondence between the keyframe’s self-attention matrix and the label, the edited keyframe information is propagated to the non-keyframes, specifically:

[0046] Set the oth frame I o The corresponding A o Each tag u in is mapped to the self-attention calculation result A of the kth frame k The corresponding marks in are as follows:

[0047] A o [u]=A k [φ ok (u)].

[0048] On the other hand, a training-free long video editing device based on self-attention and key frames comprises:

[0049] A diffusion feature extraction module is used to extract the diffusion features of each frame of the original video using a pre-trained diffusion model;

[0050] A similarity heat map calculation module is used to calculate the similarity between frames of the original video according to the diffusion characteristics, and obtain a similarity heat map between any two frames;

[0051] The video segmentation module is used to determine the similarity of the content of each frame based on the similarity heat map, and dynamically segment the video according to the similarity to obtain multiple video clips;

[0052] A key frame selection module, used for sampling at least one key frame in each video clip;

[0053] A self-attention pruning module is used to calculate the similarity between the current query key frame and all other key frames based on the diffusion feature, obtain a similarity heat map between the current query key frame and all other key frames, select a preset number of positions with large similarity from the similarity heat map between the key frames; obtain a new key matrix and a new value matrix based on the labels corresponding to the preset number of positions; and edit the self-attention matrix of the key frame based on the query matrix, the new key matrix and the new value matrix;

[0054] The key frame information propagation module is used to calculate the similarity between key frames and non-key frames in the same video segment based on the diffusion characteristics, and obtain the most similar tags between non-key frames and key frames in the same video segment; obtain the correspondence between the tags in non-key frames and key frames based on the most similar tags; based on the correspondence between the self-attention matrix of the key frames and the tags, propagate the edited key frame information to the non-key frames.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] (1) The present invention proposes an adaptive attention pruning method for compressing the key and value sequences in the extended self-attention. Specifically, the diffusion feature is first used to calculate the cosine similarity heat map between key frames, and then the M pixel positions with the highest values ​​in the heat map are selected, and only the marks corresponding to these positions are retained to form a new key and value sequence. This pruning process significantly reduces the number of marks involved in the self-attention calculation, thereby reducing the computational complexity, while maintaining the coherence and consistency of the video content; the extended self-attention pruning technology of the present invention reduces the consumption of computing resources in the key frame editing process, can significantly improve the efficiency of long video editing, and is particularly suitable for processing video content with a large number of frames;

[0057] (2) The present invention also proposes a method for dynamically selecting key frames to improve the efficiency and quality of long video editing. Specifically, the similarity between frames is calculated by analyzing the diffusion characteristics of video frames, and the video is dynamically segmented based on these similarities. In each video segment, the method automatically samples key frames, which represent the most representative content in the segment. The process of dynamically selecting key frames takes into account the changes in video content, ensuring that more key frames are selected in parts with faster content changes, and fewer key frames are selected in areas with relatively stable content. This method not only improves the flexibility and controllability of video editing, but also significantly reduces the consumption of computing resources, so that long video editing tasks can be completed efficiently with limited computing resources.

[0058] (3) The present invention can achieve minute-long video editing in a single inference, editing more than one thousand frames on a single NVIDIA A800 graphics processing unit (GPU), which is about ten times longer than existing methods;

[0059] (4) The present invention can guide the video editing process through text prompts, thereby improving the flexibility and controllability of editing;

[0060] (5) The present invention is not only applicable to a specific type of video, but can also process videos of various scenes and contents, and has wide applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 Flow chart of a training-free long video editing method based on self-attention and key frames according to an embodiment of the present invention;

[0062] Figure 2 An example flow chart of a training-free long video editing method based on self-attention and key frames implemented by the present invention;

[0063] Figure 3 A schematic diagram of a key frame selection method according to an embodiment of the present invention;

[0064] Figure 4 Schematic diagram of an adaptive attention pruning method according to an embodiment of the present invention;

[0065] Figure 5 A comparison diagram of the method of the embodiment of the present invention and other video editing methods;

[0066] Figure 6 This is a structural block diagram of a training-free long video editing device based on self-attention and key frames according to an embodiment of the present invention. DETAILED DESCRIPTION

[0067] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0068] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0069] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. The embodiments described are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this invention belongs. The terms used in the description of this invention are for the purpose of describing specific embodiments only and are not intended to limit the invention.

[0071] As used in the description and claims of this invention and the above drawings, the term "comprising" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products, or devices.

[0072] See Figure 1 and Figure 2 As shown, a method for long video editing without training based on self-attention and key frames of this invention includes the following steps.

[0073] S101, Use a pre-trained diffusion model to extract the diffusion features of each frame of the original video.

[0074] For the original video with a total of n frames, each frame is represented as I i (0 ≤ i ≤ n), use the pre-trained text-to-image diffusion model ∈ θ to extract the corresponding diffusion feature F i .

[0075] S102, Calculate the similarity between each frame of the original video according to the diffusion features to obtain a similarity heat map between any two frames.

[0076] Specifically, for the p-marked f i of the diffusion feature F i p corresponding to the i-th frame and the q-marked j of the feature F corresponding to the j-th frame, first calculate the cosine similarity between f i p and as follows:

[0077]

[0078] where CS represents the calculated cosine similarity; 0 ≤ i < j ≤ n, n represents the number of frames; ||f i p || represents the norm of f i p ; represents the norm of f i p ;

[0079] Then, calculate the token q* that is most similar to the token p:

[0080]

[0081] where T j represents all the tokens corresponding to the j-th frame;

[0082] Finally, obtain the similarity heatmap H i,j , and the value at the position corresponding to the token p in the heatmap is:

[0083]

[0084] It should be noted that the token mentioned in the present invention refers to a high-dimensional (e.g., 512-dimensional) feature.

[0085] S103. Determine the similarity of the content of each frame based on the similarity heatmap, and dynamically segment the video according to the similarity to obtain multiple video segments.

[0086] Specifically, take the first frame of the video as the start of the first video segment, and sequentially calculate H 1,2 , H 1,3 , ……, H 1,m until the mean value of all points in the calculated heatmap H 1,m is less than the first threshold ms, or use a sliding window to detect the heatmap. When there is a window at a certain position and the mean value within the window is less than the second threshold ws, take the m-th frame as the start of a new video segment, and continue to calculate H m,m+1 , H m,m+2 …… until the entire video is traversed. Finally, obtain the video segmentation based on the original video. The content in each video segment is similar, while the content in different segments is quite different.

[0087] In this embodiment, ms and ws are two different thresholds, and generally ms < ws. If the mean value of all points in the entire heatmap is less than ms (indicating that the overall content of the picture has changed), or the mean value within a certain sliding window is less than ws (indicating that a relatively drastic change has occurred in a certain area of the picture), the video will be divided into a new segment.

[0088] S104. Sample at least one key frame in each video segment.

[0089] Specifically, as shown in Figure 2 and Figure 3 , at each denoising step t of the diffusion model, in the video that has been segmented, randomly select one frame from each segment as the key frame.

[0090] S105, calculate the similarity between the current query key frame and all other key frames based on the diffusion feature, obtain the similarity heat map between the current query key frame and all other key frames, select a preset position with large similarity from the similarity heat map between the key frames; based on the labels corresponding to the preset positions, obtain a new key matrix and a new value matrix; based on the query matrix, the new key matrix and the new value matrix, edit and obtain the self-attention matrix of the key frame.

[0091] The self-attention matrix of the key frame is expressed as follows:

[0092]

[0093] Among them, A k represents the self-attention matrix of the k-th key frame; Attention represents the self-attention function; Q k represents the query matrix; represents the new bond matrix; Represents the new value matrix.

[0094] See also Figure 2 and Figure 4 As shown, in this embodiment, computing resource consumption is reduced by retaining the tags in the key and value sequences that are most relevant to the query frame. For each key frame, the following steps are performed: the cosine similarity heat map of this key frame and all other key frames is calculated using the diffusion feature, and the calculation method of the similarity heat map is the same as that implemented in S102. From these heat maps, the M positions with the highest values ​​are selected (that is, for each query, the tag tokens corresponding to the positions with the highest M values ​​are retained to participate in the attention calculation). For the keys and values ​​in the extended self-attention, only the tags corresponding to these M positions are retained, and new keys and values ​​are obtained, whose lengths are much shorter than the default.

[0095] S106, calculating the similarity between key frames and non-key frames in the same video segment based on the diffusion features, obtaining the most similar tags between non-key frames and key frames in the same video segment; obtaining the correspondence between tags in non-key frames and key frames based on the most similar tags; based on the correspondence between the self-attention matrix of the key frames and the tags, propagating the edited key frame information to the non-key frames.

[0096] Given an original video and an obtained video segment, the correspondence between each non-key frame in the same segment and the mark in the key frame in this segment is calculated, and the similarity calculation method S102 between the key frame and the non-key frame in the same video segment is implemented.

[0097] Let keyframe be I k , non-key frame is I o , based on cosine similarity, obtain non-key frame I oThe spatial position mark u corresponds to the key frame I k The most similar spatial position is marked as v * , non-keyframe I o The spatial position mark u and key frame I k The corresponding relationship of the spatial position mark v* is expressed as follows:

[0098] φ ok (u) = v *

[0099] For each non-key frame I o , there is a key frame I in the same video clip k , the attention matrix A of the oth frame in the calculation of the extended self-attention o Each tag in the kth frame is mapped to the attention matrix A in the calculation of the extended self-attention k The corresponding mark in can be expressed as:

[0100] A o [u]=A k [φ ok (u)].

[0101] Furthermore, for each time step t of the diffusion process, the above S104 to S106 are repeated until the entire diffusion process is completed and a high-quality edited video is generated.

[0102] The method of the present invention is compared with other video editing methods as follows.

[0103] Figure 5 The figure intuitively shows the performance comparison between the long video editing method of the present invention and other existing video editing methods. This figure shows the effect of different methods on processing the same video editing task through a series of carefully designed experiments, thus highlighting the superiority of the method of the present invention.

[0104] exist Figure 5 In FIG. 1 , the top row shows some frames of the original video, which represent the original state of the video content. The following rows show the results after being processed using different video editing methods. Each row corresponds to a specific editing method, including the method of the present invention and several existing video editing techniques.

[0105] The frames in each row show different stages of the editing process, or video content at different points in time. Figure 5 The detailed effects of each method in processing video content are clearly demonstrated, including object changes, background adjustments, and overall style transfer.

[0106] In the application of the method of the present invention, it can be observed that the coherence and consistency of the video content are well maintained. Whether it is changing the main object in the video, adjusting the background, or converting the overall style, the method can accurately edit according to the text prompts while ensuring that the parts of the video that are not mentioned in the edit are consistent with the source video.

[0107] In contrast, other editing methods may have some problems when processing long videos, such as flickering in some parts of the image, object deformation, or background inconsistency. These problems are usually related to high computing resource consumption and low editing efficiency, while the method of the present invention effectively solves these problems through adaptive attention pruning and key frame selection technology.

[0108] Overall, Figure 5 It not only demonstrates the superior performance of the present invention in long video editing tasks, but also highlights its unique advantages in processing complex scenes and maintaining video coherence. These comparative results prove the innovation and practicality of the present invention's method in the field of video editing, especially in application scenarios that require processing a large number of frames and maintaining high editing quality.

[0109] See also Figure 6 As shown, the present invention also discloses a training-free long video editing device based on self-attention and key frames, comprising:

[0110] Diffusion feature extraction module 601, used to extract diffusion features of each frame of the original video using a pre-trained diffusion model;

[0111] A similarity heat map calculation module 602 is used to calculate the similarity between frames of the original video according to the diffusion feature to obtain a similarity heat map between any two frames;

[0112] The video segmentation module 603 is used to determine the similarity of the content of each frame based on the similarity heat map, and dynamically segment the video according to the similarity to obtain multiple video clips;

[0113] A key frame selection module 604 is used to sample at least one key frame in each video clip;

[0114] The self-attention pruning module 605 is used to calculate the similarity between the current query key frame and all other key frames based on the diffusion feature, obtain a similarity heat map between the current query key frame and all other key frames, select a preset number of positions with large similarity from the similarity heat map between the key frames; obtain a new key matrix and a new value matrix based on the labels corresponding to the preset number of positions; and edit the self-attention matrix of the key frame based on the query matrix, the new key matrix and the new value matrix;

[0115] The key frame information propagation module 606 is used to calculate the similarity between the key frames and non-key frames in the same video segment based on the diffusion characteristics, and obtain the most similar tags between the non-key frames and the key frames in the same video segment; obtain the correspondence between the tags in the non-key frames and the key frames based on the most similar tags; based on the correspondence between the self-attention matrix of the key frames and the tags, propagate the edited key frame information to the non-key frames.

[0116] A specific implementation of a training-free long video editing system based on self-attention and key frames. The same training-free long video editing method based on self-attention and key frames will not be repeated in this embodiment.

[0117] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "comprising" does not exclude other units or, and the singular does not exclude the plural. Multiple units or systems stated in the system claim can also be implemented by one unit or system through software or hardware.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A training-free long video editing method based on self-attention and keyframes, characterized in that: include: S101, using the pre-trained diffusion model to extract the diffusion features of each frame of the original video; S102, calculating the similarity between frames of the original video according to the diffusion feature, and obtaining a similarity heat map between any two frames; S103, determining the similarity of the contents of each frame based on the similarity heat map, and dynamically segmenting the video according to the similarity to obtain multiple video segments; S104, sampling at least one key frame in each video segment; S105, calculating the similarity between the current query key frame and all other key frames based on the diffusion feature, obtaining a similarity heat map between the current query key frame and all other key frames, and selecting a preset number of positions with large similarity from the similarity heat map between the key frames; Based on the marks corresponding to the preset positions, a new key matrix and a new value matrix are obtained; Based on the query matrix, the new key matrix and the new value matrix, the self-attention matrix of the key frame is edited; S106, calculating the similarity between the key frame and the non-key frame in the same video segment according to the diffusion feature, obtaining the most similar tags between the non-key frame and the key frame in the same video segment; obtaining the corresponding relationship between the tags in the non-key frame and the key frame based on the most similar tags; Based on the self-attention matrix of keyframes and the correspondence between the tags, the edited keyframe information is propagated to non-keyframes.

2. The training-free long video editing method based on self-attention and key frames according to claim 1 is characterized in that: Use the pre-trained diffusion model to extract the diffusion features of each frame of the original video, including: For a total of n frames of the original video, each frame is represented by I i , using the pre-trained text-to-image diffusion model ∈ θ Extract the diffusion feature F corresponding to each frame i ; where 0≤i≤n.

3. The training-free long video editing method based on self-attention and key frames according to claim 1 is characterized in that: The similarity between the frames of the original video is calculated based on the diffusion features, and the similarity heat map between any two frames is obtained, which specifically includes: For the diffusion feature F corresponding to the i-th frame i The p mark f i p And for the j-th frame feature F j The q mark First calculate f i p and The cosine similarity is as follows: Among them, CS represents the calculated cosine similarity; 0 ≤ i < j ≤ n, where n represents the number of frames; ||f i p || represents the norm of f i p ; represents the norm of f i p ; Then calculate the mark q* that is most similar to mark p: Where T j Represents all the tags corresponding to the jth frame; Finally, we get the similarity heat map H i,j , the value of the position corresponding to the mark p in the heat map for:

4. The training-free long video editing method based on self-attention and key frames according to claim 1 is characterized in that: Based on the similarity heat map, the similarity of the content of each frame is determined, and the video is dynamically segmented according to the similarity to obtain multiple video clips, including: Take the first frame of the video as the beginning of the first video segment, and calculate the similarity heat map H between the first frame and the subsequent frames in sequence 1,2 , H 1,3 ,……,H 1,m , when the mean value of the points in the designated area calculated based on the similarity heat map is less than a preset threshold, the first frame to the m-1th frame are taken as the first video segment; Take the mth frame as the start of a new video clip and calculate the similarity heat map H between the mth frame and subsequent frames in sequence m,m+1 , H m,m+2 ... until the mean value of the points in the specified area calculated based on the similarity heat map is less than a preset threshold value, segmentation is performed; The entire video is traversed according to the above segmentation method to obtain multiple video clips.

5. The training-free long video editing method based on self-attention and key frames according to claim 4 is characterized in that: The mean value of the points in the specified area is less than the preset threshold, including: The mean of all points in the similarity heat map is less than the first threshold, or, when the similarity heat map is detected using a sliding window, when there is a window at a certain position, the mean of all points in the window is the second threshold.

6. The training-free long video editing method based on self-attention and key frames according to claim 1 is characterized in that: At least one key frame is sampled in each video clip, including: At each denoising step t of the diffusion model, a frame is randomly selected from each video segment in the segmented video as a key frame.

7. The training-free long video editing method based on self-attention and key frames according to claim 1 is characterized in that: The self-attention matrix of the key frame is expressed as follows: Among them, A k represents the self-attention matrix of the k-th key frame; Attention represents the self-attention function; Q k represents the query matrix; represents the new bond matrix; Represents the new value matrix.

8. The training-free long video editing method based on self-attention and key frames according to claim 1 is characterized in that: In S106, let the key frame be I k , non-key frame is I o , based on cosine similarity, obtain non-key frame I o The spatial position mark u corresponds to the key frame I k The most similar spatial position is marked as v * , non-keyframe I o The spatial position mark u and key frame I k The corresponding relationship of the spatial position mark v* is expressed as follows: φ ok (U)=v * Based on the correspondence between the keyframe’s self-attention matrix and the label, the edited keyframe information is propagated to the non-keyframes, specifically: Set the oth frame I o The corresponding A o Each tag u in is mapped to the self-attention calculation result A of the kth frame k The corresponding marks in are as follows: A o [u]=A k [φ ok (u)]。 9. A training-free long video editing device based on self-attention and key frames, characterized in that: include: A diffusion feature extraction module is used to extract the diffusion features of each frame of the original video using a pre-trained diffusion model; A similarity heat map calculation module is used to calculate the similarity between frames of the original video according to the diffusion characteristics, and obtain a similarity heat map between any two frames; The video segmentation module is used to determine the similarity of the content of each frame based on the similarity heat map, and dynamically segment the video according to the similarity to obtain multiple video clips; A key frame selection module, used for sampling at least one key frame in each video clip; A self-attention pruning module is used to calculate the similarity between the current query key frame and all other key frames based on the diffusion feature, obtain a similarity heat map between the current query key frame and all other key frames, and select a preset position with large similarity from the similarity heat map between the key frames; Based on the marks corresponding to the preset positions, a new key matrix and a new value matrix are obtained; Based on the query matrix, the new key matrix and the new value matrix, the self-attention matrix of the key frame is edited; The key frame information propagation module is used to calculate the similarity between the key frame and the non-key frame in the same video segment according to the diffusion characteristics, obtain the most similar tags between the non-key frame and the key frame in the same video segment; and obtain the corresponding relationship between the tags in the non-key frame and the key frame based on the most similar tags; Based on the self-attention matrix of keyframes and the correspondence between the tags, the edited keyframe information is propagated to non-keyframes.

Images