Zero-shot video editing method and system based on adaptive segmentation and cross-fragment interaction

Abstract

The application belongs to the technical field of video editing, and particularly relates to a zero-sample video editing method and system based on adaptive segmentation and cross-segment interaction, which comprises the following steps: encoding a source video to a latent space and extracting features to construct an initial noise latent variable; adaptively dividing semantic segments and selecting representative anchor frames based on feature similarity; screening redundant Tokens in a spatiotemporal neighborhood based on the anchor frames and performing soft weighted merging, and synchronously constructing a mapping index table; alternately switching the merged Token set in a double mode in the diffusion later stage, and performing cross-segment joint self-attention calculation based on the index table to enhance global continuity; and finally, iteratively denoising by a collaborative attention mechanism to generate an edited video with time sequence structure preserved. The application realizes time sequence structure preservation, cross-segment semantic coherence and calculation efficiency improvement in long video editing, and can complete high-quality video editing without model fine-tuning.

Description

Technical Field

[0001] This invention belongs to the field of video editing technology, specifically relating to a zero-sample video editing method and system based on adaptive segmentation and cross-segment interaction. Background Technology

[0002] With the significant progress made by diffusion models in text-guided image generation and editing, researchers have begun to extend their capabilities to video editing tasks. However, the high cost and computational demands of constructing high-quality video-to-text pairing datasets hinder the widespread adoption of large-scale text-to-video (T2V) diffusion model training. Furthermore, the flexibility of pre-trained text-to-video diffusion models in practical applications is limited by the diversity of video resolution, duration, and dynamic content changes. Therefore, current mainstream research focuses on adapting pre-trained text-to-image diffusion models to handle video data. This zero-shot video editing paradigm eliminates the need for fine-tuning of pre-trained models. Instead, it leverages existing mature text-to-image (T2I) diffusion models (such as Stable Diffusion) to modify visual content based on target text cues while preserving the spatial structure and motion information of the source video. Existing techniques primarily enhance inter-frame temporal consistency by introducing strategies such as temporal attention mechanisms and motion modeling modules, which has become a research hotspot in the field of video generation and editing.

[0003] Despite the progress made by existing methods, significant shortcomings remain. First, the semantic adaptability of current technologies is relatively rigid. Strategies such as video frame grouping and token processing often employ fixed parameter designs, failing to adapt to dynamic semantic changes in video content, such as differences in scene switching frequency and changes in object motion complexity. This leads to the loss of core semantics or redundant computation. Second, in complex semantic editing scenarios, the alignment accuracy between text and video content is limited, making it difficult to balance editing accuracy with the protection of source information. More significantly, existing technologies lack effective cross-segment semantic interaction mechanisms. When long videos are segmented, abrupt transitions between segments and inconsistent object attributes severely impair the overall smoothness and narrative coherence of the video. Finally, there is a sharp contradiction between the inherently high computational requirements of diffusion models and the efficiency of long video processing. Existing methods lack efficient long video adaptation strategies, often requiring segmented editing followed by splicing, which is not only inefficient but also prone to continuity distortion, limiting the practical application of the technology in ordinary hardware environments. Summary of the Invention

[0004] The purpose of this invention is to provide a zero-shot video editing method and system based on adaptive segmentation and cross-segment interaction, so as to solve the problems of long video temporal structure breakage, cross-segment semantic incoherence and low computational efficiency in existing zero-shot video editing technologies.

[0005] The present invention achieves the above objectives through the following technical solutions:

[0006] Firstly, this invention proposes a zero-shot video editing method based on adaptive segmentation and cross-segment interaction, including:

[0007] The source video frame sequence is encoded into a latent variable sequence, and the DIFT features of each frame are extracted. DDIM inversion is performed on the latent variable sequence to construct the initial noise latent variables.

[0008] The semantic similarity between frames is calculated based on DIFT features. The source video is divided into multiple semantic segments according to the similarity threshold, and a representative anchor frame is selected for each semantic segment.

[0009] Using the anchor frames of each segment as semantic benchmarks, redundant tokens are filtered out in the preset spatiotemporal neighborhood and merged into the tokens corresponding to the anchor frames according to similarity weights. At the same time, an index table of mapping coordinates and weights is constructed to obtain the merged token set of each segment.

[0010] In the later time step of diffusion, the merged token set is combined alternately according to a preset dual mode, and cross-segment joint self-attention calculation is performed on the combined tokens based on the index table to obtain the feature sequence;

[0011] The feature sequence and target text prompts are semantically fused under a cross-attention mechanism. The initial noise latent variables are updated through iterative denoising, and the edited video with temporal structure preserved is generated by decoding.

[0012] Furthermore, the step of encoding the source video frame sequence into a latent variable sequence, extracting the DIFT features of each frame, and performing DDIM inversion on the latent variable sequence to construct the initial noise latent variables includes:

[0013] The source video frame sequence is obtained, and each video frame is encoded from the pixel space to a low-dimensional latent space through a pre-trained variational autoencoder to obtain a sequence of latent variables.

[0014] The intermediate layer of the U-Net decoder, based on a pre-trained text-image diffusion model, extracts the DIFT features corresponding to the latent variables of each frame at diffusion time step t=0, resulting in the DIFT feature sequence. ,in C is the number of feature channels, and H and W are the height and width of the feature map, respectively;

[0015] The DDIM inversion strategy is used to transform the latent variable sequence into a noisy latent variable sequence as the initial noisy latent variable.

[0016] Furthermore, the resulting multiple semantic segments include:

[0017] Based on the DIFT feature sequence F, for any adjacent frames and Semantic similarity heatmaps are obtained through token-level cosine similarity calculation. ;

[0018] Set similarity threshold Traverse the video frame sequence and calculate the relationship between the current frame l and the starting frame of the current segment. Heatmap mean , For the first The starting index of each segment;

[0019] like If so, mark the current frame as the starting point of the new segment and update the starting index. Otherwise, the current frame is added to the current segment, and this process continues until all frames are traversed, ultimately resulting in M ​​variable-length semantic segments, and the starting index set of the segments is recorded. .

[0020] Furthermore, the selection of representative anchor frames includes:

[0021] For the m-th semantic segment, it contains One video frame, The DIFT features of each frame within the segment are averaged along the spatial dimension to obtain frame-level feature vectors. , ,in To extract from DIFT feature sequences The one extracted from the global position is located at the 1st position. Frame characteristics;

[0022] The feature matrix for constructing frame-level feature vectors ,in For the m-th semantic segment, the segment in playback order is... The frame-level feature vector corresponding to the frame; calculate the feature matrix. cosine similarity matrix The element value in the i-th row and j-th column is... Calculate the frame correlation score for each frame. The frame with the highest score is selected as the anchor frame for that segment. To form an anchor frame set To characterize the temporal structure of the source video.

[0023] Furthermore, the step of using the anchor frames of each segment as semantic benchmarks, filtering redundant tokens within a preset spatiotemporal neighborhood, and merging them into the tokens corresponding to the anchor frames based on similarity weights includes:

[0024] The target token set is composed of all tokens in each semantic fragment anchor frame. As a semantic benchmark for token merging, For anchor frames The spatial coordinates of the token on the anchor frame feature map;

[0025] The spatiotemporal neighborhood is defined spatially by setting a neighborhood window centered on the target token, and temporally by using overlay anchor frames. The windows of the frames before and after it; for the first Any frame in the segment of Only the intersection of the frames and the token within the spatiotemporal neighborhood is retained as the merging candidate set;

[0026] The L1 distance is used as the similarity metric, as shown in the following formula:

[0027] ;

[0028] in, For the source to be evaluated For the source token in the 1st The value of the dimensional feature; Candidates for comparison with the source token as a candidate In the Specific eigenvalues ​​in the dimension;

[0029] Set similarity threshold Filter to meet The tokens are highly redundant, forming the final candidate set W for merging;

[0030] A soft-weighted fusion strategy dominated by anchor frame semantics is used for token merging. After merging... As shown in the following formula:

[0031] ;

[0032] The anchor frame token weight is fixed at 1, and the candidate token weight is its similarity coefficient. A mapping index table is generated synchronously. ,in Record the current Corresponding anchor frame Token coordinates ; Stores the current token's combined weight coefficient.

[0033] Furthermore, obtaining the merged token set of each fragment includes:

[0034] An iterative strategy centered on the anchor frame is used for token merging. First, the core window consisting of the anchor frame and its direct neighbors is merged. Then, the merged result is used as a basis to propagate to other frames in the segment level by level. After the merged token is updated by the U-Net self-attention module, it maintains the merged state and enters the cross-segment interaction stage. After completion, token splitting is performed based on the mapping index table M to restore the original number of tokens.

[0035] Furthermore, in the later time step of diffusion, the merged token set is combined alternately according to a preset dual-mode, and cross-segment joint self-attention calculation is performed on the combined tokens based on the index table to obtain a feature sequence, including:

[0036] Cross-segment interaction is initiated when t < τ in the later time step of diffusion denoising, where τ is a preset initiation time step threshold; in consecutive later denoising time steps, the following two segment combination modes are executed alternately:

[0037] Mode 1: Uses a pairwise neighborhood combination method of (clip1, clip2), (clip3, clip4), ...;

[0038] Mode 2: Using a sliding neighborhood combination method of (clip1), (clip2, clip3), (clip4, clip5), ...;

[0039] Extract the merged token set of the combined segments and input it into the U-Net self-attention module to perform joint self-attention calculation; reuse the generated mapping index table M, and locate the corresponding relationship of the same token between adjacent segments through the pre-stored token coordinate mapping relationship and weight coefficient in the table;

[0040] After cross-segment joint self-attention computation, the updated Token set containing cross-segment semantic information is used as the feature sequence.

[0041] Furthermore, the step of semantically fusing the feature sequence and target text prompts under a cross-attention mechanism, updating the initial noise latent variables through iterative denoising, and decoding to generate a temporally structure-preserving edited video includes:

[0042] The simplified tokens, after merging the adaptive tokens from the segments, are input into the self-attention module in a structured form, which strengthens the local consistency between frames within the segment; before entering the cross-attention module, the structured tokens are split into the original ordered frame sequence based on the mapping index table M.

[0043] With initial noise latent variable Taking the target text hint embedding C' as input, the DDIM sampling and denoising process is initiated. Each denoising step follows the inverse update formula of the latent diffusion model, as follows:

[0044] ;

[0045] in, The noise prediction network is built into the model. For the noise scheduling parameters at time step t, Let be the noisy latent variable at time step t. The latent variables at time step t-1 after the denoising step;

[0046] After iterative denoising, the edited clean latent variable sequence is obtained. The pre-trained VAE decoder is used to inversely map the data back to pixel space, generating an edited video frame sequence. .

[0047] Secondly, this invention proposes a zero-shot video editing system based on adaptive segmentation and cross-segment interaction, used to implement the zero-shot video editing method described above. The system includes:

[0048] The preprocessing module is used to encode the source video frame sequence into a latent variable sequence, extract the DIFT features of each frame, and perform DDIM inversion on the latent variable sequence to construct the initial noise latent variables;

[0049] The temporal structure analysis module is used to calculate the inter-frame semantic similarity based on the DIFT features, divide the source video into multiple semantic segments according to the similarity threshold, and select a representative anchor frame for each semantic segment.

[0050] The segment optimization module is used to filter redundant tokens in a preset spatiotemporal neighborhood based on the anchor frame of each segment, and merge them into the tokens corresponding to the anchor frame according to similarity weights. Simultaneously, an index table of mapping coordinates and weights is constructed to obtain the merged token set of each segment.

[0051] The cross-segment interaction module is used to alternately combine the merged token set according to a preset dual mode in the later time step of the diffusion denoising process, and perform cross-segment joint self-attention calculation on the combined tokens based on the index table to enhance the semantic coherence between segments.

[0052] The video generation module is used to collaboratively execute self-attention, cross-segment interaction and cross-attention mechanisms to iteratively denoise and reconstruct the initial noise latent variables, and finally decode and generate an edited video that preserves the temporal structure.

[0053] The beneficial effects of this invention are as follows:

[0054] This invention accurately captures and preserves the temporal structure and narrative logic of the source video through adaptive video segmentation and anchor frame selection, overcoming the semantic adaptation rigidity problem caused by fixed grouping strategies in existing methods. Combined with an adaptive token merging strategy within segments, highly redundant tokens are filtered and fused within the spatiotemporal neighborhood, significantly reducing computational complexity and memory usage in long video processing, achieving a balance between efficiency and accuracy. By designing a dual-mode alternation of video segments, dynamic combination and joint self-attention calculation of adjacent segments are performed in the later stages of diffusion, effectively enhancing cross-segment semantic interaction and feature coherence, solving the problems of abrupt segment transitions and poor overall continuity in long video editing. Finally, under zero-shot conditions, the attention mechanism of the pre-trained model is collaboratively utilized to achieve accurate response to target text prompts and complete preservation of the temporal structure of the source video. Attached Figure Description

[0055] Figure 1 A flowchart illustrating the zero-shot video editing method based on adaptive segmentation and cross-segment interaction provided by this invention;

[0056] Figure 2 This is a schematic diagram of the data preprocessing method provided by the present invention;

[0057] Figure 3 A schematic diagram of the adaptive video segmentation and anchor frame selection process provided by the present invention;

[0058] Figure 4 This is a schematic diagram of the intra-segment adaptive token merging process provided by the present invention;

[0059] Figure 5 This is a schematic diagram illustrating the alternation of two combination modes of adjacent segments provided by the present invention;

[0060] Figure 6 This is a schematic diagram of the video generation, editing, and decoding process provided by the present invention;

[0061] Figure 7 This is a system block diagram of the zero-sample video editing system based on adaptive segmentation and cross-segment interaction provided by the present invention. Detailed Implementation

[0062] The present application will now be described in further detail with reference to the accompanying drawings. It should be noted that the following specific embodiments are only used to further illustrate the present application and should not be construed as limiting the scope of protection of the present application. Those skilled in the art can make some non-essential improvements and adjustments to the present application based on the above application content.

[0063] This disclosure proposes a zero-shot video editing method based on adaptive segmentation and cross-segment interaction. It accurately captures the temporal structure of the source video through adaptive video segmentation and anchor frame selection, and filters redundant tokens using an adaptive token merging strategy, achieving efficient inference and core semantic preservation in long videos while significantly reducing reliance on computational resources. Simultaneously, relying on the native attention architecture of a pre-trained T2I diffusion model, it enhances intra-segment frame consistency through self-attention and achieves text-image semantic alignment through cross-attention. A dynamic neighborhood combination strategy is designed to achieve cross-segment semantic interaction through alternating dual modes, improving the global continuity of long videos. In the later time step of diffusion, token merging features and cross-segment interaction features are processed collaboratively, fully preserving the narrative logic and rhythm of the source video while accurately responding to target text editing instructions, achieving a balance between temporal structure preservation and editing effect.

[0064] Please combine Figure 1 Specifically, the zero-shot video editing method based on adaptive segmentation and cross-segment interaction includes the following steps:

[0065] S1. Source Video Preprocessing and Latent Space Mapping

[0066] The source video frame sequence is encoded into a latent variable sequence, and the DIFT features of each frame are extracted. DDIM inversion is then performed on the latent variable sequence to construct the initial noise latent variables.

[0067] Please combine Figure 2 In a preferred embodiment, step S1 includes:

[0068] S101. Video Frame Sequence Acquisition and Feature Extraction

[0069] Input source video (n is the total number of frames), the frame resolution is uniformly adjusted to H×W (e.g., 384×672) (where H is the video frame height and W is the video frame width), and each frame is processed by a pre-trained VAE encoder (Variational Autoencoder). Encoding from pixel space to low-dimensional latent space yields a clean sequence of latent variables. The encoding process achieves 8x downsampling, with latent variable dimensions of [B, 4, H / 8, W / 8], where B is the batch size. The intermediate layers of the 2D U-Net decoder, based on a pre-trained T2I diffusion model (such as Stable Diffusion v1.5), extract the diffusion features (DIFT features) at time t=0 for each frame. ,in (C represents the number of feature channels), used for subsequent inter-frame semantic similarity calculation.

[0070] S102. Latent Variable DDIM Inversion

[0071] The DDIM (Denoising Diffusion Implicit Models) inversion strategy is used to transform the clean latent variable sequence. Transform into noisy latent variables (T is the total diffusion time step, set to 50 by default), constructing the initial noise seed for generating the edited video. The inversion process follows the DDIM forward diffusion formula:

[0072] ;

[0073] (in For noise scheduling parameters, (The noise is Gaussian noise), and the noise injection is completed through T iterations.

[0074] As an example, the video resolution is set to 384×672 (H=384, W=672), the frame rate is 25fps, and the single video duration is 240 frames (total number of frames n=240); the video frame dimension is (240, 3, 384, 672), representing 3 channels, a frame size of 384×672, and a total of 240 frames; Stable Diffusion 1.5 (SD 1.5) is used as the core T2I diffusion model, and the DDIM scheduler is used for sampling and inversion, with a sampling step size T=50. The ordered sequence of 240 frames from the source video is extracted. Each frame's pixel values ​​are normalized to the [0,1] range; each frame is then processed by a VAE encoder. Encode to a low-dimensional latent space, achieve 8x downsampling, and the latent variable dimension is ([B=8, 4, 48, 84]), resulting in a clean latent variable sequence. Extract DIFT features for each frame from the intermediate layer of the 2D U-Net decoder in SD 1.5 (time t = 0). The feature dimensions are [C=512, 384, 672] (number of channels × height × width), used for subsequent inter-frame semantic similarity calculation. The calculation is completed in 50 iterations using the DDIM forward diffusion formula. To noisy latent variables inversion, As the initial noise seed for editing.

[0075] S2. Adaptive Video Segmentation and Anchor Frame Selection

[0076] The semantic similarity between frames is calculated based on DIFT features. The source video is divided into multiple semantic segments according to the similarity threshold, and a representative anchor frame is selected for each semantic segment.

[0077] Please combine Figure 3In a preferred embodiment, step S2 includes:

[0078] S201. Semantic-driven adaptive video segmentation

[0079] Calculate inter-frame semantic similarity heatmap based on DIFT features: for any adjacent frames and Heatmaps are obtained through token-level cosine similarity calculation. Quantify the semantic differences between two frames. Set a similarity threshold. Traverse the video frame sequence and calculate the mean heatmap value between the current frame and the starting frame of the current segment (Mean( ()( (where m is the starting index of the m-th segment): If Mean( )< Then mark the current frame as the starting point of the new segment and update... Otherwise, the current frame is added to the current segment, and this process continues until all frames are traversed, ultimately resulting in M ​​variable-length semantic segments, with the segment starting index set as follows. ;

[0080] As an example, when the heatmap mean (Mean) )< At that time, the current frame is marked as the starting point of a new segment. Finally, the 240 frames of video are divided into 9 variable-length semantic segments, with the segment starting index set S={1,28,55,82,109,136,163,190,217}, and the segment length ranges from 25 to 30 frames, accurately corresponding to the 4 semantic switching boundaries of the source video.

[0081] S202. Adaptive selection of representative anchor frames

[0082] For the m-th segment, containing Frame (taking the 4th segment as an example, including) =27 frames), which contains For each video frame, the DIFT features of each frame are averaged along the spatial dimension to obtain a frame-level feature vector. :

[0083] ;

[0084] in To extract from DIFT feature sequences The one extracted from the global position is located at the 1st position. Frame characteristics;

[0085] The feature matrix for constructing frame-level feature vectors ( ),in For the m-th semantic segment, the segment in playback order is... The frame-level feature vector corresponding to the frame; calculate the feature matrix. cosine similarity matrix ( ), where the element value in the i-th row and j-th column is ,in Let i be the frame-level feature vector of the m-th semantic segment in the i-th frame. Let j be the frame-level feature vector of the m-th semantic segment; calculate the frame relevance score for each frame. ( The frame with the highest score is selected as the anchor frame for that segment. To form an anchor frame set To characterize the temporal structure of the source video.

[0086] As an example, the frame with the highest score is selected as the anchor frame; in this example, frame 14 is used as the anchor frame. =82+14-1=95. Similarly, the anchor frame set A={15,42,68,95,122,149,176,203,230} of the 9 segments is obtained, forming the temporal structure representation of the source video.

[0087] In a preferred embodiment, step S3 includes:

[0088] S3. Intra-fragment adaptive token merging

[0089] Using the anchor frames of each segment as semantic benchmarks, redundant tokens are filtered out within a preset spatiotemporal neighborhood and merged into the tokens corresponding to the anchor frames according to similarity weights. Simultaneously, an index table of mapping coordinates and weights is constructed to obtain the merged token set of each segment.

[0090] Please combine Figure 4 In a preferred embodiment, step S3 includes:

[0091] S301. Token Merging Basis and Spatiotemporal Window Constraints

[0092] Let all the tokens of the anchor frames of each segment constitute the target token set. Using anchor frame semantics as the core benchmark, the direction of token merging within a segment is constrained. A fixed Spatial Temporal window is designed: a 3×3 neighborhood window is used in the spatial dimension (covering the current token and its 8 surrounding neighboring tokens); a window with a radius of k=1 is used in the temporal dimension (covering the anchor frame and one frame before and after, i.e., [ For the first segment (m=1), the time window is replaced with... For the last segment (m=M), the time window is replaced with... Mirror-filled windows are used to reduce computational redundancy by leveraging the local semantic continuity of the video. For frames at the edge of segments, mirror-filled windows are used to avoid noise interference introduced by zero-filling and ensure the consistency of window structure across all frames.

[0093] S302. Redundant Token Filtering and Similarity Calculation

[0094] Token for any frame f in the m-th segment Only the intersection of tokens within the Spatial window and the Temporal window is retained as the candidate set for merging, excluding semantically irrelevant tokens. L1 distance (i.e., Manhattan distance) is used as a lightweight similarity metric to calculate the similarity coefficient between the current token and candidate tokens. The formula is as follows:

[0095] ;

[0096] in, For the source to be evaluated For the source token in the 1st The value of the dimensional feature; Candidates for comparison with the source token as a candidate In the Specific eigenvalues ​​in the dimension;

[0097] Set similarity threshold ,filter The highly redundant tokens form the final candidate set W for merging.

[0098] Those skilled in the art will understand that in step S302, the L1 distance similarity metric is only a preferred implementation method. This metric can be replaced by other functions that can effectively quantify the difference or similarity between two high-dimensional feature vectors, such as cosine similarity, Euclidean distance (L2 distance), Mahalanobis distance, etc.

[0099] S303. Token Soft Merging and Mapping Index Table Construction

[0100] A soft-weighted fusion strategy dominated by anchor frame semantics is adopted to merge candidate tokens to obtain the merged tokens. The formula is as follows:

[0101] ;

[0102] The anchor frame token has a fixed weight of 1, while the candidate token has a weight equal to its similarity coefficient sim, ensuring that the core semantics of the segment are not lost. A mapping index table is generated synchronously. : Record the current token Corresponding anchor frame Token coordinates ; Store the current token's merging weight coefficient to provide structural support for subsequent token splitting and cross-segment interactions.

[0103] As an example, consider the token (x=20, y=30) in frame 90 (f=90) of segment 4, if mapped to the anchor frame. If (x'=20, y'=30), then M[90,20,30,0]=(95,20,30), and M[f,x,y,1] stores the merging weight coefficients, providing support for subsequent token splitting and cross-segment interaction. This step reduces the computational cost of merging single-segment tokens from... Second drop Second-rate.

[0104] S304. Iterative Diffusion Merging and Token Splitting

[0105] When performing the merging, an iterative strategy centered on the anchor frame is adopted: first, the core 3-frame window consisting of the anchor frame and its immediate neighbors is processed. The merge is performed, and then the merged result is used as a basis to propagate to other frames within the segment, ensuring that each frame's token participates in the merge operation only once, achieving efficient semantic transmission. After the merged token is updated by the U-Net self-attention module, token splitting is performed based on the mapping index table M to restore the original number of tokens. After the merged tokens are updated by the U-Net self-attention module, they remain in the merged state and enter the cross-segment interaction stage S4. After S4 is completed, token splitting is performed uniformly based on the mapping index table M to restore the complete time sequence for use by Cross-Attention.

[0106] S4. Alternating two combination patterns for adjacent segments

[0107] In the later time step of diffusion, the merged token set is combined alternately according to a preset dual mode, and cross-segment joint self-attention calculation is performed on the combined tokens based on the index table to obtain the feature sequence.

[0108] Please combine Figure 5 In a preferred embodiment, step S4 includes:

[0109] S401. Combination Timing and Dual-Mode Alternation Design

[0110] This step achieves interaction between adjacent segments through dynamic neighborhood combination, specifically by alternating between two combination modes, only occurring in the later stages of diffusion. Immediately afterwards, cross-segment interaction is initiated, followed by Cross-Attention calculation. By this time, the diffusion process has completed semantic contour shaping, focusing on segment transition optimization during the detail rendering stage. A dynamic alternation strategy for odd and even time steps is designed:

[0111] A dual-mode late-stage time step alternation strategy is designed to achieve comprehensive cross-segment semantic interaction through complementary coverage of two combined modes:

[0112] Mode 1 (Basic Neighborhood Pairing): Using a pairwise neighborhood combination method of (clip1, clip2), (clip3, clip4), ..., direct semantic association between adjacent segments is constructed to ensure the smoothness of basic transition and the accuracy of feature alignment.

[0113] Mode 2 (Sliding Neighborhood Connection): This mode uses a sliding neighborhood combination method (clip1), (clip2, clip3), (clip4, clip5), ... to expand the coverage of cross-segment interactions, strengthen the semantic association between non-contiguous adjacent segments (such as clip2 and clip3, clip4 and clip5), and avoid local semantic breaks. The dual-mode combination is executed alternately in the later time step of diffusion, and the dynamic complementarity of the combination form realizes the full-scene semantic interaction of adjacent segments. This ensures both the semantic independence of each segment and the global coherence and natural transition of long video editing, solving the problem of incomplete semantic coverage caused by fixed combination forms.

[0114] As an example, the dual-mode alternating execution initiates cross-segment interaction mode one (basic neighborhood pairing) at 10 time steps in the later stage of diffusion (t=41 to t=50): when t=41, 43, 45, 47, 49, using pair combinations of (clip1, clip2), (clip3, clip4), ..., (clip8, clip9) to construct direct semantic associations between adjacent segments. Mode two (sliding neighborhood connection): when t=42, 44, 46, 48, 50, using sliding combinations of (clip1), (clip2, clip3), ..., (clip8, clip9) to strengthen the semantic connection between non-contiguous adjacent segments.

[0115] S402. Cross-segment joint self-attention computation

[0116] For adjacent segments after the dual-mode alternation, a simplified token set after segment adaptive token merging is extracted and input into the U-Net self-attention module to perform joint self-attention computation. This enhances the semantic correlation and structural consistency between segments through feature interaction. During the joint self-attention computation, the mapping index table M generated in the segment adaptive token merging stage is reused. With the help of the pre-stored token coordinate mapping relationship and weight coefficients in the table, the correspondence between homologous tokens between adjacent segments is quickly located, which significantly reduces the computational cost of cross-segment feature matching, while ensuring the accuracy and efficiency of cross-segment interaction and avoiding additional computational consumption.

[0117] S5. Video Generation, Editing, and Decoding

[0118] The feature sequence and target text prompts are semantically fused under a cross-attention mechanism. The initial noise latent variables are updated through iterative denoising, and the edited video with temporal structure preserved is generated by decoding.

[0119] Please combine Figure 6 In a preferred embodiment, step S5 includes:

[0120] S501. Attention Mechanism Coordination

[0121] Based on a pre-trained T2I diffusion model (such as Stable Diffusion 1.5), this invention deeply integrates its native Self-Attention and Cross-Attention core modules. Leveraging the model's built-in attention architecture, and combined with the token processing and cross-segment combination strategies designed in this invention, it optimizes the preservation of the source video's temporal structure and editing consistency, maintaining zero-sample characteristics throughout. During diffusion denoising, latent variables interact collaboratively according to the following logic to ensure semantic coherence and editing accuracy: The simplified tokens, after segment adaptive token merging, are input into the Self-Attention module in a structured organization form, segment by segment. Through spatial semantic interaction between tokens, local consistency between frames within a segment is strengthened, while the semantic dominance of anchor frames stabilizes the core semantic expression of the segment. Before entering the Cross-Attention module, the structured tokens are split into the original ordered frame sequence. This step maintains zero-sample characteristics throughout, requiring no fine-tuning of the pre-trained model.

[0122] S502. Iterative Denoising and Editing Generation

[0123] Noisy latent variables obtained by DDIM inversion Embedding target text prompts As input, initiate the DDIM sampling and denoising process, iterating for T = 50 steps to complete the denoising process. Each denoising step strictly follows the inverse update formula of the latent diffusion model:

[0124] ;

[0125] in The model's built-in noise prediction network outputs predicted noise to update the noisy latent variables at the current time step t. ; The latent variables at time step t-1 after the denoising step; The noise scheduling parameter decreases with time step t to ensure the stability and convergence of the denoising process. This is the noise scheduling coefficient corresponding to time step t-1.

[0126] The target text prompt is embedded The text descriptions used to guide video editing are fixed-dimensional semantic vector representations obtained after being transformed by a pre-trained text encoder (such as CLIP's text encoder).

[0127] S503. Latent Variable Decoding and Video Reconstruction

[0128] After iterative denoising, the edited clean latent variable sequence is obtained. The pre-trained VAE decoder inversely maps the low-dimensional latent space back to the pixel space, generating an edited video frame sequence that matches the size of the source video frames. The final output is a zero-sample edited video V' with the same frame rate as the source video and a complete temporal structure.

[0129] The video editing method disclosed herein is primarily aimed at scenarios with high demands for maintaining the temporal structure and efficient editing of videos. For example, in the field of long-form video creative editing, it can be applied to the secondary creation and semantic adjustment of movie clips, vlogs, and advertising videos, achieving style transfer and object replacement while maintaining the original video's narrative rhythm and scene logic. In terms of video restoration and adaptation, it is suitable for the image quality restoration and content enhancement of old or damaged video materials, restoring visual quality while fully preserving the original video's temporal structure and semantic coherence. Furthermore, this method also supports batch video customization, efficiently processing batch content such as e-commerce promotional videos and social media short videos, achieving unified style transfer and key information embedding across multiple videos, significantly improving editing efficiency while ensuring that the independent temporal structure and semantic integrity of each video are not compromised.

[0130] Please combine Figure 7Based on the same inventive concept, another specific embodiment of this disclosure proposes a zero-shot video editing system based on adaptive segmentation and cross-segment interaction, which is used to implement the zero-shot video editing method as described in the above embodiment. The system includes a preprocessing module 10, a temporal structure analysis module 20, an intra-segment optimization module 30, a cross-segment interaction module 40, and a video generation module 50.

[0131] The preprocessing module 10 is used to encode the source video frame sequence into a latent variable sequence, extract the DIFT features of each frame, and perform DDIM inversion on the latent variable sequence to construct the initial noise latent variables;

[0132] The temporal structure analysis module 20 is used to calculate the inter-frame semantic similarity based on the DIFT features, divide the source video into multiple semantic segments according to the similarity threshold, and select a representative anchor frame for each semantic segment.

[0133] The segment optimization module 30 is used to filter redundant tokens in a preset spatiotemporal neighborhood using the anchor frame of each segment as the semantic benchmark, and merge them into the tokens corresponding to the anchor frame according to similarity weights. Simultaneously, an index table of mapping coordinates and weights is constructed to obtain the merged token set of each segment.

[0134] The cross-segment interaction module 40 is used to alternately combine the merged token set according to a preset dual mode in the later time step of the diffusion denoising process, and perform cross-segment joint self-attention calculation on the combined tokens based on the index table to enhance the semantic coherence between segments.

[0135] The video generation module 50 is used to collaboratively execute self-attention, cross-segment interaction and cross-attention mechanisms to iteratively denoise and reconstruct the initial noise latent variables, and finally decode and generate an edited video that preserves the temporal structure.

[0136] Optionally, the system also includes a video input interface for receiving source video data streams or video files; and an output interface for outputting the generated edited video to a storage device or display device.

[0137] It should be noted that each module in the above video editing system corresponds to a step in implementing the above video editing method. Multiple modules and their corresponding steps have the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments.

[0138] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the zero-sample video editing methods described above.

[0139] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform the steps of any of the zero-sample video editing methods described above.

[0140] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0141] In addition, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0142] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A zero-shot video editing method based on adaptive segmentation and cross-segment interaction, characterized in that, include: The source video frame sequence is encoded into a latent variable sequence, and the DIFT features of each frame are extracted. DDIM inversion is performed on the latent variable sequence to construct the initial noise latent variables. The semantic similarity between frames is calculated based on DIFT features. The source video is divided into multiple semantic segments according to the similarity threshold, and a representative anchor frame is selected for each semantic segment. Using the anchor frames of each segment as semantic benchmarks, redundant tokens are filtered out in the preset spatiotemporal neighborhood and merged into the tokens corresponding to the anchor frames according to similarity weights. At the same time, an index table of mapping coordinates and weights is constructed to obtain the merged token set of each segment. In the later time step of diffusion, the merged token set is combined alternately according to a preset dual mode, and cross-segment joint self-attention calculation is performed on the combined tokens based on the index table to obtain the feature sequence; The feature sequence and target text prompts are semantically fused under a cross-attention mechanism. The initial noise latent variables are updated through iterative denoising, and the edited video with temporal structure preserved is generated by decoding.

2. The zero-shot video editing method based on adaptive segmentation and cross-segment interaction according to claim 1, characterized in that, The process of encoding the source video frame sequence into a latent variable sequence, extracting the DIFT features of each frame, and performing DDIM inversion on the latent variable sequence to construct the initial noise latent variables includes: The source video frame sequence is obtained, and each video frame is encoded from the pixel space to a low-dimensional latent space through a pre-trained variational autoencoder to obtain a sequence of latent variables. The intermediate layer of the U-Net decoder, based on a pre-trained text-image diffusion model, extracts the DIFT features corresponding to the latent variables of each frame at diffusion time step t=0, resulting in the DIFT feature sequence. ,in C is the number of feature channels, and H and W are the height and width of the feature map, respectively; The DDIM inversion strategy is used to transform the latent variable sequence into a noisy latent variable sequence as the initial noisy latent variable.

3. The zero-shot video editing method based on adaptive segmentation and cross-segment interaction according to claim 2, characterized in that, The resulting multiple semantic segments include: Based on the DIFT feature sequence F, for any adjacent frames and Semantic similarity heatmaps are obtained through token-level cosine similarity calculation. ; Set similarity threshold Traverse the video frame sequence and calculate the relationship between the current frame l and the starting frame of the current segment. Heatmap Mean ), Let be the starting index of the m-th segment; If Mean( ) < If so, mark the current frame as the starting point of the new segment and update the starting index. Otherwise, the current frame is added to the current segment, and this process continues until all frames are traversed, ultimately resulting in M ​​variable-length semantic segments, and the starting index set of the segments is recorded. .

4. The zero-shot video editing method based on adaptive segmentation and cross-segment interaction according to claim 3, characterized in that, The selection of representative anchor frames includes: For the m-th semantic segment, it contains One video frame, The DIFT features of each frame within the segment are averaged along the spatial dimension to obtain frame-level feature vectors. , ,in To extract from DIFT feature sequences The one extracted from the global position is located at the 1st position. Frame characteristics; The feature matrix for constructing frame-level feature vectors ,in For the m-th semantic segment, the segment in playback order is... The frame-level feature vector corresponding to the frame; calculate the feature matrix. cosine similarity matrix The element value in the i-th row and j-th column is... Calculate the frame correlation score for each frame. The frame with the highest score is selected as the anchor frame for that segment. To form an anchor frame set To characterize the temporal structure of the source video.

5. The zero-shot video editing method based on adaptive segmentation and cross-segment interaction according to claim 4, characterized in that, The step of using the anchor frames of each segment as semantic benchmarks, filtering redundant tokens within a preset spatiotemporal neighborhood, and merging them into the tokens corresponding to the anchor frames based on similarity weights includes: The target token set is composed of all tokens in each semantic fragment anchor frame. As a semantic benchmark for token merging, For anchor frames The spatial coordinates of the token on the anchor frame feature map; The spatiotemporal neighborhood is defined spatially by setting a neighborhood window centered on the target token, and temporally by using overlay anchor frames. The window of each frame before and after it; the token of any frame f in the m-th segment. Only the intersection of the frames and the token within the spatiotemporal neighborhood is retained as the merging candidate set; The L1 distance is used as the similarity metric, as shown in the following formula: ； in, For the source token to be evaluated, For the source token in the 1st The value of the dimensional feature; Candidate Tokens to be compared with the source Token as a candidate In the Specific eigenvalues ​​in the dimension; Set similarity threshold Filter to meet The tokens are highly redundant, forming the final candidate set W for merging; A soft-weighted fusion strategy dominated by anchor frame semantics is used for token merging. The merged tokens are... As shown in the following formula: ； The anchor frame token weight is fixed at 1, and the candidate token weight is its similarity coefficient. A mapping index table is generated synchronously. ,in Record the current Corresponding anchor frame Token coordinates ; Stores the current token's combined weight coefficient.

6. The zero-shot video editing method based on adaptive segmentation and cross-segment interaction according to claim 5, characterized in that, The obtained merged token set of each fragment includes: An iterative strategy centered on the anchor frame is used for token merging. First, the core window consisting of the anchor frame and its direct neighbors is merged. Then, the merged result is used as a basis to propagate to other frames in the segment level by level. After the merged token is updated by the U-Net self-attention module, token splitting is performed based on the mapping index table M to restore the original number of tokens.

7. The zero-shot video editing method based on adaptive segmentation and cross-segment interaction according to claim 5, characterized in that, In the later time step of diffusion, the merged token set is combined alternately according to a preset dual-mode, and cross-segment joint self-attention calculation is performed on the combined tokens based on the index table to obtain a feature sequence, including: Cross-segment interaction is initiated when t < τ in the later time step of diffusion denoising, where τ is a preset initiation time step threshold; in consecutive later denoising time steps, the following two segment combination modes are executed alternately: Mode 1: Uses a pairwise neighborhood combination method of (clip1, clip2), (clip3, clip4), ...; Mode 2: Using a sliding neighborhood combination method of (clip1), (clip2, clip3), (clip4, clip5), ...; Extract the merged token set of the combined segments and input it into the U-Net self-attention module to perform joint self-attention calculation; reuse the generated mapping index table M, and locate the corresponding relationship of the same token between adjacent segments through the pre-stored token coordinate mapping relationship and weight coefficient in the table; After cross-segment joint self-attention computation, the updated Token set containing cross-segment semantic information is used as the feature sequence.

8. The zero-shot video editing method based on adaptive segmentation and cross-segment interaction according to claim 1, characterized in that, The step of semantically fusing the feature sequence and target text prompts under a cross-attention mechanism, updating the initial noise latent variables through iterative denoising, and decoding to generate a temporally structure-preserving edited video includes: The simplified tokens, after merging the adaptive tokens from the segments, are input into the self-attention module in a structured form, which strengthens the local consistency between frames within the segment; before entering the cross-attention module, the structured tokens are split into the original ordered frame sequence based on the mapping index table M. With initial noise latent variable Taking the target text hint embedding C' as input, the DDIM sampling and denoising process is initiated. Each denoising step follows the inverse update formula of the latent diffusion model, as follows: ; in, The noise prediction network is built into the model. For the noise scheduling parameters at time step t, Let be the noisy latent variable at time step t. For the latent variables at time step t-1 after the denoising step, where Embed for target text hints; After iterative denoising, the edited clean latent variable sequence is obtained. The pre-trained VAE decoder is used to inversely map the data back to pixel space, generating a sequence of edited video frames. .

9. A zero-shot video editing system based on adaptive segmentation and cross-segment interaction, used to implement the zero-shot video editing method as described in any one of claims 1-8, characterized in that, The system includes: The preprocessing module encodes the source video frame sequence into a latent variable sequence, extracts the DIFT features of each frame, and performs DDIM inversion on the latent variable sequence to construct the initial noise latent variables. The temporal structure analysis module is used to calculate the inter-frame semantic similarity based on the DIFT features, divide the source video into multiple semantic segments according to the similarity threshold, and select a representative anchor frame for each semantic segment. The segment optimization module is used to filter redundant tokens in a preset spatiotemporal neighborhood based on the anchor frame of each segment, and merge them into the tokens corresponding to the anchor frame according to similarity weights. Simultaneously, an index table of mapping coordinates and weights is constructed to obtain the merged token set of each segment. The cross-segment interaction module is used to alternately combine the merged token set according to a preset dual mode in the later time step of the diffusion denoising process, and perform cross-segment joint self-attention calculation on the combined tokens based on the index table to enhance the semantic coherence between segments. The video generation module is used to collaboratively execute self-attention, cross-segment interaction and cross-attention mechanisms to iteratively denoise and reconstruct the initial noise latent variables, and finally decode and generate an edited video that preserves the temporal structure.

Images