Training-free long video generation method based on long frame and short frame principal component analysis

Abstract

The invention relates to the technical field of video generation, and discloses a training-free long video generation method based on long frame and short frame principal component analysis, which comprises the following steps: adding noise in a video frame to obtain initial noise; iteratively denoising the initialized noise through a denoising network; slicing long frame features through a sliding window to obtain short frame features; enabling the long-frame video and the short-frame video to pass through a time sequence attention module respectively, and outputting global features and local features; projecting to a principal component space, selecting the first k components with the highest similarity as consistent appearance features through cosine similarity calculation, and taking the remaining components as exercise intensity features; splicing the decoupled consistent appearance features and the motion intensity features in the sliding window to obtain fusion features; the fused features are subjected to noise prediction and sampler updating to obtain noise; and inputting the noise into a decoder to obtain a recovered video frame. According to the method and the device, the generated long video can have consistency and good quality at the same time by utilizing the different advantages of the long frame feature and the short frame feature.

Description

Technical Field

[0001] The present invention relates to the technical field of video generation, and in particular to a training-free long video generation method based on principal component analysis of long frames and short frames. Background Art

[0002] In the field of video generation, the quality of video generation continues to improve with the development of diffusion model technology. Existing video generation models are mostly trained on short-frame video datasets. Once long-frame videos are generated, they suffer from quality degradation, semantic loss, and slow motion. To achieve high-quality long-frame video generation using pre-trained video diffusion models, a number of non-training methods have emerged, improving the quality of long video generation. Some research has segmented the long video generation process into segments using a sliding window, then used a designed fusion algorithm to splice the short segments into a long video.

[0003] While the above methods are quite effective, the sliding window format disrupts the consistency of the video generation process, resulting in drastic temporal jumps in the generated video. Some studies have proposed directly generating long-frame videos and then extracting quality-related information from short-frame video features to supplement the long-frame video features. This approach avoids the video inconsistency caused by the sliding window, thereby generating high-quality, consistent long-frame videos. However, directly generating long-frame videos inevitably deviates from the distribution of the original training data, making it difficult to further improve the quality. In short, how to effectively utilize long-frame and short-frame video features is crucial for generating highly consistent, high-quality long-frame videos.

[0004] To this end, the present invention proposes a method for generating long videos without training based on principal component analysis of long frames and short frames. Summary of the Invention

[0005] In order to solve the above technical problems, the present invention provides a long video generation method based on principal component analysis of long frames and short frames without training.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] A method for generating long videos without training based on principal component analysis of long and short frames, comprising the following steps:

[0008] S1, add noise to the video frame to get the initial noise Z T ;

[0009] S2, iteratively denoise the initial noise through the denoising network of the diffusion model. In each denoising step, the denoising network performs the following operations until the iteration index t = 0:

[0010] The noise Z t The features obtained after convolution and spatial attention processing are used as long-frame features, which are sliced ​​into short-frame features through a sliding window. The long-frame video and short-frame video are respectively passed through the temporal attention module to output global features and local features. The global features and local features are projected into the principal component space, and the top k components with the highest similarity are selected from the global features by calculating the cosine similarity as the consistent appearance features. The components in the local features corresponding to the consistent appearance features are removed, and the remaining components are used as motion intensity features.

[0011] The decoupled consistent appearance features and motion intensity features in the sliding window are spliced, and the proportion of consistent appearance features is gradually increased as the sliding window sequence number increases during fusion, and mapped back to the original space to obtain the fused features; the fused features are updated through noise prediction and sampler to obtain the noise Z of the tth iteration t ; t = T-1;

[0012] S3, input the noise Z0 into the decoder to obtain the restored video frame.

[0013] In one embodiment, noise is added to the video frame to obtain the initial noise Z T , specifically including:

[0014] An initial noise mean reuse process is designed to extract the mean of the initial noise of the original video frame and use it to replace the mean of the initial noise of the subsequently added video frames. The other parts of the subsequently added video frames are sampled from the Gaussian distribution.

[0015] In one embodiment, the design of the initial noise mean reuse process, extracting the mean of the initial noise of the original video frame to replace the mean of the initial noise of the subsequently added video frame, specifically includes:

[0016] Extract the mean of the initial f frames of noise in the video and replace it with the mean of the initial noise of the remaining Ff frames, where F is the total number of frames in the video. Then, perform a random ordering operation on the video frames after noise replacement:

[0017]

[0018] ∈′ j:j+f =∈ j:j+f -mean(∈ j:j+f )+mean(∈ 1:f );

[0019] Where,∈ τ represents the initial noise at the τth time frame, τ represents the index of the current time frame, It is represented as a Gaussian distribution with a mean of 0 and a variance of 1. To implement the mean multiplexing operation grouped at intervals of the number of frames f, let j = nf + 1, and j < F - f, where mean represents the operation of taking the mean; ∈′ j:j+f represents the set of initial noises after the mean multiplexing operation from the j-th time frame to the j + f-th time frame, and j:j + f represents from the j-th time frame to the j + f-th time frame, ∈ j:j+f represents the set of original initial noises from the j-th time frame to the j + f-th time frame;

[0020] The finally obtained initial noises are represented after arrangement as:

[0021] [∈1, ∈2, …, ∈ f , sh(∈′ f+1:2f+1 ), …, sh(∈′ j:j+f ), …];

[0022] In the formula, sh represents the operation of randomly arranging the video frame sequence. <00​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​The transfer matrix calculated in principal component analysis is denoted as P. The global features and local features are transformed into the principal component space in the time dimension using principal component analysis through the transfer matrix. It can be specifically expressed by the following formula:

[0030]

[0031] in, Represents the operation of the collection and distribution transfer matrix in principal component analysis, It is the corresponding feature of global features and local features in the principal component space;

[0032] In one embodiment, the method of calculating the cosine similarity by selecting the top k components with the highest similarity from the global features as the consistent appearance features, removing the components corresponding to the consistent appearance features in the local features, and using the remaining components as the motion intensity features specifically includes:

[0033] Calculate the principal component space and The cosine similarity between The top k components with the largest similarity are used as consistency features And will The components corresponding to the consistency feature are removed and used as motion intensity features

[0034]

[0035] n (1) ,n (2) ,…,n (f) =argsort(s (1) ,s (2) ,…,s (f) );

[0036]

[0037] Among them, CosSim represents calculating the cosine similarity between f components, and argsort represents calculating the index of the cosine similarity arranged in ascending order.

[0038] In one embodiment, the decoupled consistent appearance features and motion intensity features within the sliding window are spliced, and the proportion of consistent appearance features is gradually increased as the sliding window sequence increases during fusion, and then mapped back to the original space to obtain fused features, specifically including:

[0039] Different k values ​​are used for different sliding windows to gradually inject consistent features. The relationship between the i-th sliding window and the k value is shown in the following formula:

[0040]

[0041] K max is the set value;

[0042] After progressive fusion, the transposed transfer matrix P is used T Map the features back to the original space:

[0043]

[0044] Among them, Concat represents the splicing operation in the time dimension. is the splicing feature; is the fusion feature.

[0045] Compared with the prior art, the beneficial technical effects of the present invention are:

[0046] Based on deep learning technology, this paper designs a fusion algorithm of long frame and short frame features to improve the quality and consistency of long video generation, and proposes the use of principal component analysis technology to extract consistency features, thereby achieving better video temporal consistency, which can be applied to multiple video pre-training models; first, a video consistency feature decoupling process is designed, the global features and local features output by the temporal attention module are projected into the principal component space, and the cosine similarity is used to select components with consistency attributes; then, a progressive fusion process of video consistent appearance features and motion intensity features is designed, and a progressive fusion method is adopted for different sliding windows to maintain the quality of the original video; finally, an initial noise mean reuse process is designed to further enhance the consistency of the video.

[0047] By leveraging the distinct advantages of long and short frame features, this method enables the generation of long videos with both consistency and high quality. Furthermore, it requires no additional training and can be applied to most pre-trained diffusion models, avoiding the time-consuming training process. Applying this method to multi-prompt video generation and video continuation tasks can also improve video consistency while maintaining good video quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flow chart of a method according to an embodiment of the present invention;

[0049] Figure 2 This is a visualization diagram of the process of decoupling consistency features using cosine similarity in an embodiment of the present invention;

[0050] Figure 3 Schematic diagram of a method for generating long videos without training based on principal component analysis of long frames and short frames according to an embodiment of the present invention;

[0051] Figure 4Detailed diagram of the progressive fusion process of video consistent appearance features and motion intensity features according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] A preferred embodiment of the present invention will be described in detail below with reference to the accompanying drawings.

[0053] like Figure 1 As shown, the present invention proposes a method for generating long videos without training based on principal component analysis of long frames and short frames, in order to improve the visual quality of long videos generated by a video diffusion model trained on short-frame video data, and maintain good temporal consistency. The proposed method for generating long videos without training based on principal component analysis of long frames and short frames is simple and effective, without the need for an additional lengthy training process. The consistency features of long frame features are extracted through principal component analysis, and are progressively fused with short frame features in the principal component feature space, so that the generated long frame video has better details and quality, while maintaining consistency. The proposed method can be used directly on a pre-trained video diffusion model without training, and has good scalability. The present invention has significant advantages in practical applications and can be used as a general algorithm to improve video consistency.

[0054] The present invention provides a method for generating long videos without training based on principal component analysis of long and short frames. The innovations of the method are as follows:

[0055] A video consistency feature decoupling process is designed to project the features passed by the temporal attention module into the principal component space. The cosine similarities between different components of global features and local features are compared in the principal component space. Components with high similarity are selected as the consistent appearance features of the video, and components with low similarity are selected as the motion intensity features of the video.

[0056] A progressive fusion process is designed for video consistent appearance features and motion intensity features. After extracting the consistent appearance features of long-frame videos and the motion intensity features of short-frame videos, during the fusion process, the proportion of consistent appearance features is gradually increased as the number of frames increases. The original visual quality is not changed within the initial frame number range, and a smooth transition is achieved.

[0057] An initial noise mean reuse process is designed to extract the mean of the initial noise of the original frame and replace it with the mean of the initial noise of the subsequently added video frames. The rest is sampled from a Gaussian distribution, thereby maintaining the randomness of the generation process while ensuring the consistency of the generated content.

[0058] Preferably, the video consistency feature decoupling process includes the following steps:

[0059] like Figure 3As shown, first, denote the features input to the temporal attention module as where b, h, w, F, and c represent the batch size, feature height, feature width, number of feature frames, and number of channels respectively. The pre-trained model is trained on short video data with f frames, where f < F, and F is the number of video frames of the long video to be generated. Denote the global features extracted from the entire video sequence of x as and the local features extracted using the i-th sliding window as Since the length of the former is greater than that of the latter, in order to keep the feature sizes consistent, perform slicing operations on x global , which can be specifically expressed by the following formula:

[0060]

[0061] where Temp represents the temporal attention module, and Slice i represents slicing the video sequence using the i-th window.

[0062] Denote the transfer matrix calculated in the principal component analysis as Using this matrix, perform principal component analysis on the above two types of features in the time dimension to transform them into the principal component space, which can be specifically expressed by the following formula:

[0063]

[0064] where, represents the operation of the scatter transfer matrix in the principal component analysis, and have been re-normalized to which are the features in the calculated principal component space.

[0065] To extract the consistency features, calculate the cosine similarity between the two types of features in the principal component space, and take the top k components with the largest similarity in as the consistency features, and remove the components of this consistency feature in as the motion intensity features. It can be specifically expressed as:

[0066]

[0067] n (1) ,n (2) ,…,n (f) =argsort(s (1) ,s (2) ,…,s (f) );

[0068]

[0069] Among them, CosSim represents calculating the cosine similarity between f components, and argsort represents calculating the index of the cosine similarity arranged in ascending order.

[0070] Preferably, the progressive fusion process of the video consistent appearance feature and motion intensity feature is as follows: Figure 4 As shown in the figure, after obtaining the consistent appearance features of the long-frame video and the motion intensity features of the short-frame video, different k values ​​are used for different sliding windows during the fusion process to gradually inject the consistent features. Specifically, the following steps are performed:

[0071] Different k values ​​are used for different sliding windows to gradually inject consistent features. Specifically, the relationship between the i-th window and the k value is as follows:

[0072]

[0073] Among them, in order to maintain the quality of the original video, K max =3.

[0074] After progressive fusion, the transposed transfer matrix P is used T Mapping the features back to the original space can be expressed as:

[0075]

[0076] Among them, Concat represents the splicing operation in the time dimension. In order to obtain the final video sequence, the overlapping parts of the sliding windows are averaged to obtain the final x fuse The above operations are only used in the first 25 denoising steps to maintain the generated quality.

[0077] Preferably, the initial noise mean reuse process includes:

[0078] Since the temporal mean of the initial noise can reflect the appearance characteristics of the video, the mean of the initial f frames of noise is extracted and replaced with the mean of the remaining Ff frames. At the same time, in order to maintain the randomness of the initial noise, the replaced noise is randomly arranged in order. The process can be expressed as:

[0079]

[0080] ∈′ j:j+f =∈ j:j+f -mean(∈ j:j+f )+mean(∈ 1:f );

[0081] Where, j = nf + 1, And j < F - f, where mean represents the operation of taking the mean. The finally obtained initial noise can be expressed as:

[0082] [∈1, ∈2, …, ∈ f , sh(∈′ f+1:2f+1 ), …, sh(∈′ j:j+f ), …];

[0083] In the formula, sh represents the operation of randomly arranging the video frame sequence. The above process can further enhance the consistency of the video generation result.

[0084] The present invention is applied to the field of video generation technology, and proposes a method for generating long videos without training based on principal component analysis of long frames and short frames. First, a video consistency feature decoupling process is designed. As Figure 3 shown, project the features passing through the temporal attention module into the principal component space, compare the cosine similarities between different components of the long frame features and the short frame features in the principal component space, select the components with high similarity as the consistent appearance features of the video, and select the components with low similarity as the motion intensity features of the video. The visualization diagram of the extracted features is as Figure 2 shown. Then, a progressive fusion process of the video consistent appearance features and the motion intensity features is designed. After obtaining the consistent appearance features of the long frame video and the motion intensity features of the short frame video, fuse the two. During the fusion process, gradually increase the proportion of the consistent appearance features as the number of frames increases, without changing the original visual quality within the initial number of frames, and achieve a smooth transition. Specifically, as Figure 4 shown. Finally, an initial noise mean reuse process is designed. As Figure 3 shown, extract the mean of the initial noise of the original frame and replace it with the mean of the initial noise of the subsequently added video frames. The other parts are sampled from the Gaussian distribution, so as to maintain the randomness in the generation process while ensuring the consistency of the generated content. The overall framework is as Figure 3 shown.[[ID=*]] [[ID=*]]

[0085] The method for generating long videos without training based on principal component analysis of long frames and short frames proposed by the present invention can simultaneously maintain the consistency and quality during the long video generation process, and has improvements in object consistency, background consistency, overall consistency, action smoothness, dynamic diversity, and picture quality. In the Vbench evaluation metrics, when generating video segments using 326 prompts, there are improvements in all evaluation metrics.

[0086] ​

[0087] The method for generating long videos without training based on principal component analysis of long frames and short frames proposed in the present invention can be used in video continuation. It requires an inverted sampling process and retains the initial noise, so that existing video clips can be used to generate longer video clips while maintaining the consistency of the entire video.

[0088] In summary, the present invention proposes a method for generating long videos without training based on principal component analysis of long and short frames. Consistency features are extracted from video features, and a consistency feature decoupling process is proposed, using cosine similarity to extract consistency features. A progressive fusion process is designed to gradually increase the proportion of consistency features as the window slides to ensure video consistency without sacrificing quality. In addition, an initial noise mean reuse process is introduced to further enhance video consistency. The present invention significantly outperforms existing models, achieves high fidelity and consistency, and establishes a training-free paradigm for consistency enhancement in other video generation applications.

[0089] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times. The order of execution of these steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.

[0090] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0091] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. It is intended that all variations within the meaning and range of equivalents of the claims be embraced herein, and any reference signs in the claims should not be construed as limiting the claims to which they relate.

[0092] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A training-free long video generation method based on principal component analysis of long and short frames, characterized in that: The following steps are involved: S1, add noise to the video frame to get the initial noise Z T ; S2, iteratively denoise the initialization noise through the denoising network. In each denoising step, the denoising network performs the following operations until the iteration index t = 0: The noise Z t The features obtained after convolution and spatial attention processing are used as long-frame features, which are sliced ​​into short-frame features through a sliding window. The long-frame video and short-frame video are respectively passed through the temporal attention module to output global features and local features. The global features and local features are projected into the principal component space, and the top k components with the highest similarity are selected from the global features by calculating the cosine similarity as the consistent appearance features. The components in the local features corresponding to the consistent appearance features are removed, and the remaining components are used as motion intensity features. The decoupled consistent appearance features and motion intensity features in the sliding window are spliced, and the proportion of consistent appearance features is gradually increased as the sliding window sequence number increases during fusion, and mapped back to the original space to obtain the fused features; the fused features are updated through noise prediction and sampler to obtain the noise Z of the tth iteration t ; t = T-1; S3, input the noise Z0 into the decoder to obtain the restored video frame.

2. The method for generating long videos without training based on principal component analysis of long and short frames according to claim 1, characterized in that: Add noise to the video frame to get the initial noise Z T , specifically including: An initial noise mean reuse process is designed to extract the mean of the initial noise of the original video frame and use it to replace the mean of the initial noise of the subsequently added video frames. The other parts of the subsequently added video frames are sampled from the Gaussian distribution.

3. The method for generating long videos without training based on principal component analysis of long and short frames according to claim 1, characterized in that: The design of the initial noise mean reuse process extracts the mean of the initial noise of the original video frame and uses it to replace the mean of the initial noise of the subsequently added video frame, specifically including: Extract the mean of the initial f frames of noise in the video and replace it with the mean of the initial noise of the remaining Ff frames, where F is the total number of frames in the video. Then, perform a random ordering operation on the video frames after noise replacement: ∈′ j:j+f =∈ j:j+f -mean(∈ j:j+f )+mean(∈ 1:f ); where ∈ τ represents the initial noise at the τ-th time frame, and τ represents the index of the current time frame. represents a Gaussian distribution with a mean of 0 and a variance of 1. To perform the mean multiplexing operation in groups at an interval of frame number f, let j = nf + 1. and j < F - f, where mean represents the operation of taking the mean; ∈′ j:j+f represents the set of initial noises after the mean multiplexing operation from the j-th time frame to the j + f-th time frame, and j:j + f represents from the j-th time frame to the j + f-th time frame, ∈ j:j+f represents the set of original initial noises from the j-th time frame to the j + f-th time frame; The final initial noise is expressed as follows after permutation: [∈1,∈2,…,∈ f ,sh(∈′ f+1:2f+1 ),…,sh(∈′ j:j+f ),…]; Where sh represents the operation of randomly arranging the video frame sequence.

4. The method for generating long videos without training based on principal component analysis of long and short frames according to claim 1, characterized in that: The noise Z t The features obtained after convolution operation and spatial attention processing are used as long-frame features, and the long-frame features are sliced ​​through the sliding window to obtain short-frame features; The long-frame video and the short-frame video are passed through the temporal attention module respectively, and the global features and local features are output, including: The noise Z t The feature obtained after convolution operation and spatial attention processing is recorded as x, and the global feature extracted from the entire video sequence of x is recorded as global , and the local features extracted by the i-th sliding window are recorded as In order to keep the size of global features and local features consistent, global Slice operation: Among them, Temp represents the temporal attention module, Slice i Indicates slicing the entire video sequence of x using the i-th window, Represents the global feature x global The i-th slice of .

5. The method for generating long videos without training based on principal component analysis of long and short frames according to claim 4, characterized in that: Projecting the global features and the local features into the principal component space specifically includes: The transfer matrix calculated in principal component analysis is denoted as P. The global features and local features are transformed into the principal component space in the time dimension using principal component analysis through the transfer matrix. It can be specifically expressed by the following formula: in, Represents the operation of the collection and distribution transfer matrix in principal component analysis, It is the corresponding feature between global features and local features in the principal component space.

6. The method for generating long videos without training based on principal component analysis of long and short frames according to claim 5, characterized in that: The method selects the top k components with the highest similarity from the global features as the consistent appearance features by calculating the cosine similarity, removes the components corresponding to the consistent appearance features in the local features, and uses the remaining components as the motion intensity features, specifically including: Calculate the principal component space and The cosine similarity between The top k components with the largest similarity are used as consistency features And will The components corresponding to the consistency feature are removed and used as motion intensity features n (1) ,n (2) ,…,n (f) =argsort(s (1) ,s (2) ,…,s (f) ); Among them, CosSim represents calculating the cosine similarity between f components, and argsort represents calculating the index of the cosine similarity arranged in ascending order.

7. The method for generating long videos without training based on principal component analysis of long and short frames according to claim 1, characterized in that: The decoupled consistent appearance features and motion intensity features in the sliding window are spliced, and the proportion of consistent appearance features is gradually increased as the sliding window sequence increases during fusion, and mapped back to the original space to obtain fused features, specifically including: Different k values ​​are used for different sliding windows to gradually inject consistent features. The relationship between the i-th sliding window and the k value is shown in the following formula: K max is the set value; After progressive fusion, the transposed transfer matrix P is used T Map the features back to the original space: Among them, Concat represents the splicing operation in the time dimension. is the splicing feature; is the fusion feature.

Images