A video editing method based on diffusion model correspondence

Abstract

A video editing method based on diffusion model correspondences includes the following steps: S1. Using a variational autoencoder (VAE) to extract latent features from each frame of an input video; S2. Adding noise at a specific time step to the extracted latent features; S3. Inputting the noisy latent features into a pre-trained text-to-image (T2I) diffusion model; S4. Performing an inverse denoising process in the T2I diffusion model to gradually remove the noise; S5. Extracting intermediate feature representations during the inverse denoising process to form diffusion features, which contain markers in each frame; S6. Calculating the similarity of diffusion feature markers between different frames to identify the correspondence between markers in each frame and markers in other frames; S7. Based on the identified correspondences, performing feature fusion on the identified corresponding markers, and performing video editing operations based on a self-attention mechanism on the fused features. This method can significantly improve the quality and temporal consistency of video editing.

Description

Technical Field

[0001] The present invention relates to computer vision processing technology, and in particular to a video editing method based on diffusion model correspondence. Background Art

[0002] Diffusion models have demonstrated excellent performance in image generation, thus motivating their application in the field of image editing. These methods typically leverage a pre-trained text-to-image (T2I) stable diffusion model, use DDIM inversion to convert the source image into noise, and then gradually denoise it under the guidance of hints to generate the edited image.

[0003] Despite promising performance in image editing, achieving high-quality video editing remains challenging. In particular, unlike the mature open-source T2I stable diffusion model, comparable text-to-video (T2V) diffusion models are less mature due to the difficulty in simulating complex temporal dynamics, and training a T2V model from scratch requires a lot of computational resources. Therefore, there is growing interest in applying pre-trained T2I diffusion to video editing. In this context, maintaining temporal consistency of the edited video is one of the biggest challenges, which requires the generated frames to be stylistically consistent and exhibit smooth temporal transitions rather than looking like a series of independent images. Many methods have been studying this problem, but still face various limitations, such as the inability to ensure fine temporal consistency (causing the generated videos to appear flickering or blurry), the need for additional components, or the need for additional training or optimization.

[0004] It should be noted that the information disclosed in the above background technology section is only used to understand the background of this application, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0005] The main purpose of the present invention is to overcome the defects of the above-mentioned background technology and provide a video editing method based on the corresponding relationship of the diffusion model.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A video editing method based on diffusion model correspondence relationship includes the following steps:

[0008] S1. Use variational autoencoder (VAE) to extract latent features from each frame of the input video.

[0009] S2. Add noise at a specific time step to the extracted latent features;

[0010] S3. Input the noised latent features into the pre-trained text-to-image T2I diffusion model;

[0011] S4. performing a reverse denoising process in the T2I diffusion model to gradually remove noise;

[0012] S5. Extracting intermediate feature representations during the inverse denoising process to form diffusion features, the diffusion features containing markers in each frame;

[0013] S6. Calculating the similarity of the diffusion feature markers between different frames to identify the correspondence between the markers in each frame and the markers in other frames;

[0014] S7. Based on the identified correspondence, perform feature fusion on the identified corresponding markers, and perform video editing operations based on the self-attention mechanism on the fused features.

[0015] A computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the video editing method based on the diffusion model correspondence relationship.

[0016] A computer program product includes a computer program, wherein when the computer program is executed by a processor, the video editing method based on the diffusion model correspondence relationship is implemented.

[0017] The present invention has the following beneficial effects:

[0018] The present invention provides a video editing method based on the correspondence of a diffusion model, which uses a diffusion model to obtain the correspondence of feature markers in video frames, significantly improving the quality and temporal consistency of video editing. Specifically, the present invention adds noise to the potential features extracted from the video frame and then performs the inverse denoising process of the diffusion model to extract the intermediate feature representation, forming a diffusion feature containing the markers in the frame, and utilizing the intrinsic correspondence of the markers between frames. These correspondences are closely related to the temporal consistency of the video, and by calculating the self-attention mechanism between the features corresponding to different frames, the quality and temporal consistency of video editing can be significantly improved, and the generated video can be ensured to maintain a high degree of consistency in style and content, avoiding the flickering or blurring phenomenon common in traditional methods. In addition, the present invention has the advantage of plug-and-play, and can be inserted into the pre-trained diffusion model without additional training of the existing model, thereby improving the video editing performance. This not only reduces the demand for computing resources, but also makes the video editing process more efficient, and is particularly suitable for resource-constrained scenarios such as mobile devices.

[0019] In the preferred embodiment, the present invention also proposes a method for calculating the correspondence of labels using a sliding window, which not only effectively obtains a one-to-many correspondence, but also significantly reduces the computational overhead, reduces the computational complexity, and reduces the GPU memory usage.

[0020] In summary, the present invention realizes diversified editing of video content (including background editing, foreground editing, style editing, etc.) while maintaining high-quality editing effects and superior performance, surpassing existing video editing technologies.

[0021] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 Flowchart of a video editing method based on diffusion model correspondence according to an embodiment of the present invention.

[0023] Figure 2 This is an overview diagram of the algorithm architecture of an embodiment of the present invention.

[0024] Figure 3 This is an embodiment of the present invention that calculates the correspondence between features based on a sliding window.

[0025] Figure 4 This is a practical effect diagram of an embodiment of the present invention.

[0026] Figure 5 2 is a comparison diagram of the effects of the embodiment of the present invention and the existing method. DETAILED DESCRIPTION

[0027] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope of the present invention and its application.

[0028] One goal of this invention is to achieve highly consistent video editing by leveraging the intrinsic correspondence between inter-frame markers (i.e., tokens, which represent features at a certain location in a feature map). This relationship is intuitively closely related to the temporal consistency of a video: if corresponding markers across frames show high similarity, the resulting video will exhibit high temporal consistency. For example, if the marker representing a man's nose in a video has high similarity across frames, his nose is less likely to distort or flicker in the video.

[0029] How to obtain accurate correspondence information between markers remains a less explored problem in existing research, although the intrinsic characteristics of the video editing task (i.e., the source video and the edited video are expected to share similar actions and semantic layouts) determine that such correspondences naturally exist in the source video. Some previous methods use pre-trained optical flow models to obtain the flow trajectory of each marker between frames, which can be regarded as a rough correspondence information. Although the self-attention mechanism of markers in the same trajectory can enhance the temporal consistency of the edited video, it still faces two major limitations: First, these methods heavily rely on highly accurate pre-trained optical flow models to obtain the correspondence of markers, which is not available in many scenarios. Second, assuming that an extremely accurate optical flow model is accessible, it can still only obtain a rough one-to-one correspondence between markers in different frames, which may lead to information loss because in most cases, one marker is likely to correspond to multiple markers in other frames.

[0030] To address these issues, the inventors noted that inherent diffusion features naturally contain precise correspondence information. For example, by extracting the diffusion features of two images and computing the cosine similarity between the markers, corresponding points between the two images can be easily found. However, to date, no existing work has successfully exploited this property to handle more complex and challenging tasks, such as video editing. This paper proposes a video editing method based on diffusion model correspondences, which is the first solution to leverage the potential of inherent diffusion feature correspondences to significantly improve the quality and temporal consistency of video editing.

[0031] See Figure 1 , an embodiment of the present invention provides a video editing method based on a diffusion model correspondence relationship, comprising the following steps:

[0032] S1. Use variational autoencoder (VAE) to extract latent features from each frame of the input video.

[0033] S2. Adding noise of the selected specific time step to the extracted latent features;

[0034] S3. Input the noised latent features into the pre-trained text-to-image T2I diffusion model;

[0035] S4. performing a reverse denoising process in the T2I diffusion model to gradually remove noise;

[0036] S5. Extracting intermediate feature representations during the inverse denoising process to form diffusion features, the diffusion features containing markers in each frame;

[0037] S6. Calculating the similarity of the diffusion feature markers between different frames to identify the correspondence between the markers in each frame and the markers in other frames;

[0038] S7. Based on the identified correspondence, perform feature fusion on the identified corresponding markers, and perform video editing operations based on the self-attention mechanism on the fused features.

[0039] In the video editing method based on the diffusion model correspondence of the embodiment of the present invention, the diffusion model is used to obtain the correspondence of the feature markers in the video frame, which significantly improves the quality and temporal consistency of video editing. Specifically, the present invention extracts intermediate feature representations by adding noise to the potential features extracted from the video frame and then performs the inverse denoising process of the diffusion model to form diffusion features containing the markers in the frame. The intrinsic correspondence between the markers between frames is used, and these correspondences are closely related to the temporal consistency of the video. By calculating the self-attention mechanism between the features corresponding to different frames, the quality and temporal consistency of video editing can be significantly improved, and the generated video can be ensured to maintain a high degree of consistency in style and content, avoiding the flickering or blurring phenomenon commonly seen in traditional methods.

[0040] In a preferred embodiment, step S3 further includes inputting the user's prompt words into the T2I diffusion model to guide the video editing process.

[0041] In a preferred embodiment, the T2I diffusion model includes a U-net decoder. In step S5, an intermediate feature representation output by the U-net decoder is extracted during the inverse denoising process to form a diffusion feature.

[0042] In one embodiment, step S6 specifically includes: for each marker in each frame, calculating its cosine similarity with markers in other frames; for each marker in each frame, selecting the top K markers with the highest similarity based on the cosine similarity to form a one-to-many correspondence. More specifically, it includes: normalizing the diffusion features; performing a dot product operation on the normalized diffusion features, i.e., evaluating the similarity between markers in different frames through matrix multiplication; and using the resulting similarity matrix, determining the coordinates of the top K markers in each frame with the highest similarity to the given marker.

[0043] In a preferred embodiment, in step S6, the one-to-many correspondence relationship calculation of the tags is performed based on a sliding window strategy, specifically including the following sub-steps:

[0044] S61. For each frame in the video, extract diffusion features and determine the location of the marker;

[0045] S62. For each marker in the diffusion feature, determine the center of the small window as the position of the marker in the current frame, and set the window length to limit the local range for calculating the similarity;

[0046] S63. Using the small window determined in step S62, select the marker in the small window in the next frame corresponding to it as a candidate corresponding marker;

[0047] S64. For each marker in the current frame, calculate the similarity between it and the candidate corresponding marker in the small window, and use the dot product operation to evaluate the similarity;

[0048] S65. Based on the result of the dot product operation, select the top K tags with the highest similarity to the current frame tag in the small window to determine the corresponding relationship;

[0049] S66. Repeat steps S62 to S65 to process all the marks in the current frame and determine the position of the corresponding mark for each frame.

[0050] In step S64, the dot product operation is performed by matrix multiplication of the normalized diffusion features.

[0051] The preferred embodiment uses a sliding window to calculate the corresponding relationship of the markers, which not only effectively obtains a one-to-many corresponding relationship, but also significantly reduces the computational overhead, reduces the computational complexity, and reduces the GPU memory usage.

[0052] In a preferred embodiment, step S7 specifically includes:

[0053] S71. Using the correspondence determined in step S6, sample features from the noise latent layer for each marker in each frame;

[0054] S72. Fusing the sampled features to reduce GPU memory usage and accelerate the video editing process;

[0055] S73. Apply a self-attention mechanism to the fused features, performing the following calculations: initialize the query Q, key K, and value V matrices, where Q is the input and K and V are the fused features; calculate the dot product between Q and K and normalize them using the Softmax function to obtain attention weights; perform a weighted sum of V using the obtained attention weights to obtain a weighted value vector;

[0056] S74. Edit the video frame using the result of the self-attention operation.

[0057] The method of this embodiment not only significantly improves the quality and temporal consistency of video editing, but also offers the advantage of plug-and-play functionality. It can be plugged into a pre-trained diffusion model without requiring additional training of the existing model, thereby improving video editing performance. This not only reduces the demand for computing resources but also makes the video editing process more efficient, making it particularly suitable for resource-constrained scenarios such as mobile devices.

[0058] Specific embodiments of the present invention are further described below.

[0059] In some embodiments, the video editing method includes: obtaining a source video and a prompt word; inputting the source video and prompt word into a pre-trained diffusion model to obtain an edited video; wherein the diffusion model includes a self-attention mechanism based on correspondence relationships. First, a sliding window-based calculation method is used to obtain the correspondence relationship between features of different frames. Then, during the inversion and denoising process, the self-attention mechanism is calculated based on this correspondence relationship, thereby achieving high-quality and highly temporally consistent video editing.

[0060] Given a source video, diffusion features are first extracted for each frame. Then, for each marker in the diffusion features, corresponding markers in other frames are obtained based on their similarity. In this process, a sliding window-based approach is proposed to ensure computational efficiency. In the sliding window approach, each marker only needs to calculate its similarity with the markers within a small window in the next frame, and the top K (K>1) markers with the highest similarity are identified. After the correspondence calculation process, the coordinates of the K corresponding markers in each other frame can be obtained for each marker. During the inversion and denoising process, noisy latent markers are sampled based on the obtained coordinates. To reduce redundancy and accelerate the editing process, marker merging is applied in the temporal dimension, followed by self-attention. The method can be seamlessly integrated into the existing T2I diffusion model without the need for additional training or optimization. Extensive experiments show that COVE significantly improves the quality and temporal consistency of the generated videos, surpassing a wide range of existing methods and achieving state-of-the-art results.

[0061] In this setting, the source video V contains IV frames, and a variational autoencoder (VAE) is used to extract latent features z = {x1, ...x N}, where z∈R N×H×W×d Here H and W represent the height and width of the potential features, and d represents the dimension of each tag. For each frame in z, add noise at a specific time step t to obtain the noise frame Then input them into the pre-trained text-to-image (T2I) model f θ Through the single-step operation of denoising, the diffusion features (i.e., the intermediate features of the U-net decoder) can be extracted.

[0062]

[0063] One-to-many correspondence calculation

[0064] For each mark in the diffusion feature F, its corresponding mark in other frames is determined based on the cosine similarity. Without loss of generality, we can consider the i-th frame F i With coordinates [h i , wi ]feature Unlike previous methods, which only determine a single corresponding marker per frame, this method simply obtains a one-to-many correspondence by selecting the top K markers with the highest similarity in each frame. Their coordinates are recorded and used to sample the markers in the subsequent inversion and denoising steps. This is most straightforwardly accomplished by performing matrix multiplication on the normalized diffusion features F.

[0065] s=F·F T

[0066] Features The similarity between all IV×H×W tags is given by S[i,h i , w Then, the coordinates of the corresponding features in the jth frame are obtained by selecting the top K features with the highest similarity in the jth frame (j∈{1,...,Ⅳ}).

[0067]

[0068] Here The operation represents finding the coordinates of the largest K values ​​in the matrix, where k∈{1,...,K}. Indicates the j-th frame with The coordinates of the marker with the highest similarity. A similar process can be performed for each feature in F to obtain their correspondence between frames.

[0069] Sliding window strategy

[0070] Although the one-to-many correspondence between tags can be effectively obtained through the above process, it requires a lot of computing resources because IV×H×W is always a huge number, especially in long videos. Therefore, the computational complexity of this process is extremely high, which can be expressed as O(IV 2 ×H 2 ×W 2 ×d). At the same time, the multiplication between these two huge matrices consumes a lot of GPU memory in practice. These limitations severely restrict its application in many practical scenarios, such as on mobile devices.

[0071] In order to solve the above problems, a sliding window-based strategy is further proposed as an alternative, which not only effectively obtains one-to-many correspondences but also significantly reduces the computational overhead. Just calculate its difference with the next frame F i+1 The similarity of the tags in , rather than the tags of all frames, i.e.

[0072]

[0073] in Represents the similarity of each feature between the i-th frame and the i+1-th frame. The overall similarity matrix is ​​S = {S i}, i∈{1, 2, ..., N-1} is obtained by the following formula The corresponding features in the i+1th frame:

[0074]

[0075] Similarly, the corresponding features in the i+2th frame can be obtained:

[0076]

[0077] Through the same process, the position of the corresponding feature in each frame can be obtained. Through the above process, the computational complexity is reduced to It is further noted that each frame in the video is continuous, so for It only needs to calculate the similarity within a small window of adjacent frames, rather than calculating the similarity for all features in adjacent frames.

[0078]

[0079] Represents the features within a window of length l. By only calculating the similarity within this window, the computational complexity is further reduced to In the actual operation process, the GPU memory usage is also significantly reduced. Figure 3 shown.

[0080] During the inversion and denoising process of video editing, the corresponding markers are sampled from the noise latent layer for each marker according to the coordinates obtained in the above process. The corresponding feature set in other frames at time step t is

[0081]

[0082] These features are integrated to speed up the video editing process and reduce GPU memory usage.

[0083]

[0084] Then, self-attention operation is performed on the fused features

[0085]

[0086] Through the above operations, the quality and timing consistency of video editing can be significantly improved.

[0087] The embodiment of the present invention can significantly improve the quality and temporal consistency of video editing by calculating the self-attention mechanism between the features corresponding to different frames. In the experiment, the method can perform various edits on the video content based on the prompt words provided by the user, including background editing, foreground editing, style editing, etc. For different types of prompt words, the present invention has demonstrated high editing quality. Some of the effects are as follows Figure 4 shown.

[0088] At the same time, the performance of the present invention exceeds many previous baseline methods and shows superior performance. Figure 5 shown.

[0089] In summary, the present invention proposes an innovative video editing method, which effectively solves some key problems in traditional video editing by utilizing the correspondence information contained in the diffusion model features, and significantly improves the quality and timing consistency of video editing.

[0090] Compared with the traditional method, the technical advantages of the present invention are mainly as follows:

[0091] Compared with previous video editing methods that had difficulty in ensuring the temporal consistency of generated videos, this method improves video quality and temporal consistency: by utilizing the feature correspondence in the diffusion model, it effectively improves the quality and temporal consistency of video editing, and solves problems such as video flickering.

[0092] Plug-and-play: No additional model training is required; it can be directly plugged into a pre-trained diffusion model to quickly improve video editing performance.

[0093] Efficient performance: Compared with existing similar methods, this method shows superior video editing performance.

[0094] Resource Optimization: Ability to run efficiently in resource-constrained environments, such as mobile devices, enabling high-quality video editing without requiring extensive computing resources.

[0095] The present invention has a wide range of application scenarios and high practical value, for example:

[0096] Film and television production: In the production of films and television programs, the present invention can be used to edit original filming materials, add special effects, adjust colors, and synchronize audio and video to enhance the coherence and visual appeal of the story.

[0097] Advertising Production: Video editing is used in the advertising industry to create compelling commercials that appeal to the target audience and effectively communicate product information through precise editing and visual enhancement.

[0098] Social media content creation: With the popularity of social media, this invention supports content creators in producing short videos and live broadcast content, helping them optimize video formats, add creative elements, and meet the needs of different platforms.

[0099] Education and Training: In the field of education, this invention can be used to create teaching videos and online courses, transforming complex concepts into easy-to-understand visual content and enhancing teaching effectiveness.

[0100] Video Game Development: In video game development, this invention is used to create promotional trailers and in-game story animations, involving video editing, special effects production, and sound synchronization to enhance the immersive gaming experience.

[0101] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0102] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0103] An embodiment of the present invention further provides a processor, which executes a computer program and at least performs the method described above.

[0104] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a magnetic disk memory or a magnetic tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0105] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0106] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0107] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0108] Those skilled in the art will appreciate that all or part of the steps of the above-mentioned method embodiments may be implemented by hardware associated with program instructions, and the aforementioned program may be stored in a computer-readable storage medium. When the program is executed, the program executes the steps of the above-mentioned method embodiments. The aforementioned storage medium includes various media that can store program codes, such as mobile storage devices, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0109] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0110] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0111] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0112] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0113] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art will recognize that, without departing from the scope of the present invention, several equivalent substitutions or obvious variations can be made, and the performance or use of the same should be considered to fall within the scope of protection of the present invention.

Claims

1. A video editing method based on diffusion model correspondence, characterized in that: The following steps are involved: S1. Use variational autoencoder (VAE) to extract latent features from each frame of the input video. S2. Add noise at a specific time step to the extracted latent features; S3. Input the noised latent features into the pre-trained text-to-image T2I diffusion model; S4. Performing a reverse denoising process in the T2I diffusion model to gradually remove noise; S5. Extracting intermediate feature representations during the inverse denoising process to form diffusion features, the diffusion features containing markers in each frame; S6. Calculating the similarity of the diffusion feature markers between different frames to identify the correspondence between the markers in each frame and the markers in other frames; S7. Based on the identified correspondence, perform feature fusion on the identified corresponding markers, and perform video editing operations based on the self-attention mechanism on the fused features.

2. The video editing method according to claim 1, wherein: Step S3 also includes inputting the user's prompt words into the T2I diffusion model to guide the video editing process.

3. The video editing method according to claim 2, wherein: The T2I diffusion model includes a U-net decoder. In step S5, the intermediate feature representation output by the U-net decoder is extracted during the reverse denoising process to form a diffusion feature.

4. The video editing method according to any one of claims 1 to 3, wherein: Step S6 specifically includes: for each tag in each frame, calculating its cosine similarity with the tags in other frames; for each tag in each frame, selecting the top K tags with the highest similarity based on the cosine similarity to form a one-to-many correspondence.

5. The video editing method according to claim 4, wherein: Step S6 specifically includes: normalizing the diffusion features; performing a dot product operation on the normalized diffusion features, that is, evaluating the similarity between markers in different frames through matrix multiplication; and using the obtained similarity matrix to determine the coordinates of the top K markers with the highest similarity to a given marker in each frame.

6. The video editing method according to claim 4, wherein: In step S6, the one-to-many correspondence relationship of the tags is calculated based on the sliding window strategy, which specifically includes the following sub-steps: S61. For each frame in the video, extract diffusion features and determine the location of the marker; S62. For each marker in the diffusion feature, determine the center of the small window as the position of the marker in the current frame, and set the window length to limit the local range for calculating the similarity; S63. Using the small window determined in step S62, select the marker in the small window corresponding to the next frame as a candidate corresponding marker; S64. For each marker in the current frame, calculate the similarity between it and the candidate corresponding marker in the small window, using a dot product operation to evaluate the similarity. S65. Based on the result of the dot product operation, select the top K tags with the highest similarity to the current frame tag in the small window to determine the corresponding relationship; S66. Repeat steps S62 to S65 to process all the marks in the current frame and determine the position of the corresponding mark for each frame.

7. The video editing method according to claim 6, wherein: In step S64, the dot product operation is performed by matrix multiplication of the normalized diffusion features.

8. The video editing method according to any one of claims 1 to 3, wherein: Step S7 specifically includes: S71. Using the correspondence determined in step S6, sampling features for each marker in each frame from the noise potential layer; S72. Fusion processing is performed on the sampled features to reduce GPU memory usage and accelerate the video editing process; S73. Apply the self-attention mechanism to the fused features, performing the following computations: Initialize the query Q, key K, and value V matrices, where Q is the input and K and V are the fused features; Calculate the dot product between Q and K and normalize them using the Softmax function to obtain the attention weights; Use the obtained attention weights to perform a weighted sum on V to obtain the weighted value vector; S74. Edit video frames using the results of self-attention operation.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the video editing method based on the diffusion model correspondence relationship according to any one of claims 1 to 8 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the video editing method based on the diffusion model correspondence relationship according to any one of claims 1 to 8 is implemented.

Images