A method for generating long videos with multiple texts without fine-tuning

Abstract

DuFree, a fine-tuning-free method for generating long videos with multiple texts, leverages a two-stream video diffusion model (DVDM) and designs frame-by-frame prompts and dimensionally compressed attention modules to ensure smooth transitions between multiple prompts while preserving details. The DVDM uses an expert-adaptive LayerNorm (AdaLN) module to process textual and visual inputs separately, improving content quality. The frame-by-frame prompt module maintains a prompt word queue to achieve smooth transitions between video frames, while the dimensionally compressed attention module compresses multiple prompt words, alleviating the training-inference gap and preserving texture details. This method outperforms both automatic metrics and human evaluations, particularly in tracking strength and temporal motion smoothness. It addresses the challenges of existing models in generating multiple prompts and long videos, achieving high-quality, temporally consistent, and motion-smooth long videos.

Description

Technical Field

[0001] The present invention relates to video generation technology, and in particular to a method for generating multi-text long videos without fine-tuning. Background Art

[0002] The rapid development of diffusion models has made significant progress in generative tasks, especially in the field of vision. Diffusion models, originally used for image synthesis, are able to generate highly detailed images from text descriptions through a cross-attention mechanism, demonstrating their powerful ability to model spatial content.

[0003] Building on the success of image generation, people naturally extend their interest to more complex fields, such as videos, where temporal dynamics become crucial. Therefore, many studies are devoted to applying diffusion-based methods to video generation, trying to exploit its ability to process time-series frames. Recently, a new two-stream diffusion model architecture has been proposed, which emphasizes that better alignment between text and visual content can significantly improve performance. The present invention adopts this new architectural form, namely the two-stream video diffusion model (DVDM). To achieve this, the model uses two expert adaptive LayerNorm modules to process the two modalities respectively, and the time step of the diffusion process is used as a modulation module, which significantly improves the quality of the generated content.

[0004] Despite these advances, current diffusion models for video generation still face significant challenges. Compared to images, videos are high-dimensional data, requiring larger models and more extensive training datasets. As a result, existing methods are often trained on datasets containing only a small number of video frames, limiting their ability to generate long videos. Furthermore, these methods perform poorly when generating videos containing multiple scenes or complex actions. This limitation stems from the challenges of training models using multiple cues, especially given the scarcity of video datasets containing rich, multi-stage narratives.

[0005] A straightforward approach to addressing this issue without additional training is to use a base model to generate separate video clips for each cue word and then stitch them together. This approach can create long videos containing multiple scenes with rich semantic variations. However, this often results in poor consistency between clips and discontinuous transitions. Diffusion models are very sensitive to changes in cue words, and even small changes in textual cues can lead to significant changes in the generated video content. To address this issue, FreeNoise uses cue word interpolation and injects them within a specific denoising step range to achieve video clip transitions. However, its effectiveness is limited because different frames require switching cue words at different denoising steps to achieve optimal results, and interpolation of cue word embeddings does not always produce semantically coherent transitions.

[0006] It should be noted that the information disclosed in the above background technology section is only used to understand the background of this application, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0007] The main purpose of the present invention is to overcome the defects existing in the above-mentioned background technology and provide a method for generating multi-text long videos without fine-tuning.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] A method for generating long multi-text videos without fine-tuning, comprising the following steps:

[0010] S1. Generate N frames of video using the basic model and add noise of varying intensities to each frame to simulate varying degrees of degradation, forming an initial denoising queue.

[0011] S2. Maintain a prompt word queue, where each prompt word corresponds to a video frame. Initially, the prompt word queue is filled with the first prompt word.

[0012] S3,processing text and visual inputs through the dual-stream video diffusion model DVDM, which uses two expert adaptive LayerNorm modules to process text and visual modalities respectively, and the time step of the diffusion process is used as a modulation module;

[0013] S4. During the denoising process, the prompt words are gradually switched to change the prompt words in a smoother way and interpolated in a space closer to the visual domain to achieve a smooth transition of video generation using different prompt words.

[0014] S5. Compress multiple prompt words into a single representation to alleviate the training-inference gap in DVDM while preserving texture details. Multiple prompt words are compressed into a single prompt word by averaging the prompt words over the batch dimension, which is then concatenated with the noisy image and input into the 3D full attention module to calculate the attention weights, determine which parts of the video should be matched with the text prompt, and output weighted video features.

[0015] S6. In the denoising queue, only the latent codes of the second half are updated to perform a denoising step. At the same time, the weighted video features output by the 3D full attention module are used to guide the denoising process of the latent codes. After the attention operation, the prompt word is expanded back to the original dimension so that it can be used for the final video generation. After denoising is completed, the latent code at the front of the queue is dequeued and replaced by randomly sampled Gaussian noise at the end of the queue. This cycle is repeated to generate a video of unlimited length. The weighted video features output by the 3D full attention module and the latest latent code in the denoising queue are combined with the prompt word expanded back to the original dimension to generate video frames corresponding to multiple text prompts.

[0016] S7. While the denoising queue is being updated, the prompt word queue is also updated synchronously so that each video frame corresponds to a correct text prompt.

[0017] Furthermore, in step S1, based on the basic model generating N frames of video, the DVDM is used to initialize the diagonal denoising module. By initializing the diagonal denoising module, noise of different intensities is added to each frame of video to simulate different degrees of degradation, thereby forming an initial denoising queue.

[0018] Furthermore, in step S1, initializing the denoising queue specifically includes:

[0019] When the total number of frames required is greater than the number of video frames generated by the base model at one time, additional frames are generated by reusing the first frame of the video and applying noise of varying intensities to form a complete denoising queue;

[0020] The potential codes in the denoising queue are used to initialize the denoising process and are not directly used to generate the final video frame after denoising, but are only used as the starting point of the denoising process.

[0021] Furthermore, step S4 includes:

[0022] During the denoising process, the cue words are slightly adjusted in each denoising step to achieve a smooth transition in video generation between different cue words.

[0023] Maintain a prompt word queue that matches the length of the denoising queue and iteratively update the queue to gradually switch prompt words to ensure semantic coherence between video frames.

[0024] By adjusting the coefficients before the cue words are embedded, different cue words can be applied to the same frame or different frames at different time steps, thereby achieving a smooth transition of video content in both the frame and time step dimensions.

[0025] Furthermore, in step S4, the frame-by-frame prompting and denoising synchronization specifically include:

[0026] The initial prompt word is used to fill the queue, and the corresponding video frame is not included in the final output video. During each iteration, according to the number of iterations, a new prompt word is calculated and added to the end of the queue, and the prompt word at the head of the queue is removed to maintain a one-to-one correspondence between the prompt word queue and the denoising queue.

[0027] In the denoising step, the coefficients before the cue word embedding are determined based on the position of the cue word in the current and future iterations; among them, distinguishing the cue window to be input into the model in the current and next iterations ensures the correct application of the cue word and the smooth transition of the video content.

[0028] Furthermore, the dimension compression attention processing in step S5 further includes:

[0029] After processing by the expert AdaLN module, a set of prompt words with noise intensity is obtained;

[0030] Compressing multiple prompt words into a single prompt word by averaging the prompt words in the batch dimension;

[0031] The compressed single prompt word is combined with the noisy image and input into the 3D full attention module to calculate the attention weight;

[0032] Among them, the expert AdaLN module is used to ensure the effective alignment of visual and textual spaces, and seamless transition between different prompt words is achieved by compressing the prompt words that change frame by frame.

[0033] Furthermore, in step S6, after processing by the 3D full attention module, the prompt word is expanded back to the original dimension and input into the feedforward module of the video generation model for further video frame generation processing.

[0034] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the method for generating a multi-text long video without fine-tuning.

[0035] A computer program product includes a computer program, which, when executed by a processor, implements the method for generating a multi-text long video without fine-tuning.

[0036] The present invention has the following beneficial effects:

[0037] This paper presents a tuning-free method for generating long multi-text videos, dubbed DuFree. It proposes the first tuning-free framework for generating long multi-cue words based on DVDM. This method innovatively designs frame-by-frame cueing and dimensionally compressed attention, combined with a two-stream video diffusion model (DVDM), to achieve tuning-free generation of long multi-text videos, ensuring smooth transitions between multiple cue words while preserving details. Using DVDM, the method utilizes two expert adaptive LayerNorm (AdaLN) modules to process text and visual modalities, respectively, and uses the time step of the diffusion process as a modulation module, significantly improving the quality of generated content. The frame-by-frame cueing module maintains a cue word queue that matches the denoising queue length, achieving smooth transitions between frames. The dimensionally compressed attention module compresses multiple cue words into a single representation, bridging the gap between training and inference while preserving important texture details. Experiments demonstrate that DuFree outperforms other methods in terms of segment temporal consistency, tracking strength, and temporal motion smoothness, achieving state-of-the-art results and demonstrating exceptional performance. By switching prompt words appropriately during the denoising process, this method achieves smooth transitions between video clips. This addresses the challenges faced by existing diffusion models for video generation when dealing with multiple prompt words and generating long videos, particularly when datasets are scarce. Experiments validate the effectiveness of this method, demonstrating its significant advantages in generating high-quality, temporally consistent, and motion-smooth long videos.

[0038] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is an overview of the DuFree framework of the multi-text long video generation method according to an embodiment of the present invention.

[0040] Figure 2 2 is a schematic diagram of updating the prompt queue in the frame-by-frame prompt solution according to an embodiment of the present invention.

[0041] Figure 3 This is a schematic diagram of how the frame-by-frame prompting solution according to an embodiment of the present invention takes effect.

[0042] Figure 4 3 is a comparison chart of experimental results of the embodiment of the present invention and other methods. DETAILED DESCRIPTION

[0043] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope of the present invention and its application.

[0044] It should be noted that when an element is referred to as being "fixed to" or "disposed on" another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element. In addition, connection can be used for both fixing and coupling or communication.

[0045] It should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the present invention.

[0046] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0047] This paper proposes a method for generating long videos with multiple cues without adjustment, called DuFree. It aims to leverage the advanced two-stream video diffusion model (DVDM) for inference, generating longer videos without compromising quality. Its core concept is to gradually switch cues during the denoising process, achieving smoother changes and interpolating in a space closer to the visual domain. The invention specifically incorporates two key innovations: frame-by-frame cues and dimensionally compressed attention. Its core approach is to switch cues earlier in the denoising process, applying them to later frames in each video clip, thereby increasing the number of denoising steps affected by subsequent cues. It also introduces a novel cues interpolation method to achieve smooth transitions. This is achieved through two specific modules: frame-by-frame cues and dimensionally compressed attention. Frame-by-frame cues maintain a queue of cues, with each cues corresponding to a video frame. By projecting the cues' embeddings into a visually aligned space using the DVDM's adaptive LayerNorm, the invention interpolates to obtain better visual transition values, thereby helping to guide the generation of smooth video transitions. The dimensionally compressed attention module compresses multiple cue words into one representation, alleviating the training-inference gap in DVDM when generating long videos containing multiple cue words while preserving texture details.

[0048] See Figure 1, an embodiment of the present invention provides a method for generating a multi-text long video without fine-tuning, comprising the following steps:

[0049] Step S1, initialize the denoising queue (noisy video queue): use the basic model to generate N frames of video, and add noise of different intensities to each frame to simulate different degrees of degradation to form an initial denoising queue.

[0050] In a preferred embodiment, specifically, based on the generation of N frames of video by the base model, the characteristics of the dual-stream video diffusion model (DVDM) are used to initialize the diagonal denoising module. By initializing the diagonal denoising module, noise of different intensities is added to each frame of video to simulate different degrees of degradation, thereby forming an initial denoising queue to prepare for the subsequent denoising process. Initializing the denoising queue specifically includes: when the total number of frames required is greater than the number of video frames generated by the base model at one time, additional frames are generated by reusing the first frame of the video and applying noise of different intensities to form a complete denoising queue; the latent code in the denoising queue is used to initialize the denoising process, and is not directly used to generate the final video frame after denoising, but only serves as the starting point of the denoising process.

[0051] Step S2, constructing a frame-by-frame prompt queue: maintaining a prompt word queue, each prompt word corresponds to a video frame, and the prompt word queue is initially filled with the first prompt word;

[0052] Step S3, dual-stream video diffusion model processing: using the dual-stream video diffusion model DVDM to process text and visual inputs, wherein the model uses two expert adaptive LayerNorm (AdaLN) modules to process text and visual modalities respectively, and the time step of the diffusion process is used as a modulation module to improve the quality of the generated content;

[0053] Step S4, frame-by-frame prompting and denoising synchronization: During the denoising process, the prompt words are gradually switched, the prompt words are changed in a smoother way, and interpolation is performed in a space closer to the visual domain to achieve a smooth transition of video generation using different prompt words.

[0054] In a preferred embodiment, during the denoising process, a smooth transition of video generation between different prompt words is achieved by making subtle adjustments to the prompt word in each denoising step; a prompt word queue that matches the length of the denoising queue is maintained, and the queue is iteratively updated to switch prompt words in a progressive manner to ensure semantic coherence between video frames; and by adjusting the coefficients before the prompt word is embedded, different prompt words are applied to the same frame or different frames at different time steps, thereby achieving a smooth transition of video content in both the frame and time step dimensions.

[0055] In a preferred embodiment, the synchronization of frame-by-frame prompting and denoising specifically includes: initializing a prompt word queue, wherein the initial prompt word c0 is used to fill the queue, and its corresponding video frame is not included in the final output video; during each iteration, according to the number of iterations, a new prompt word is calculated and added to the end of the queue, and the prompt word at the head of the queue is removed to maintain a one-to-one correspondence between the prompt word queue and the denoising queue; in the denoising step, the coefficient of the prompt word before embedding is determined based on the position of the prompt word in the current and future iterations; wherein, the prompt window to be input into the model in the current and next iterations is distinguished to ensure the correct application of the prompt word and the smooth transition of the video content.

[0056] Step S5, dimensionally compressed attention processing: compress multiple prompt words into one representation to alleviate the training-inference gap in DVDM while preserving texture details; wherein, by averaging the prompt words on the batch dimension, N prompt words are compressed into a single prompt word, which is then concatenated with the noisy image and input into the 3D full attention module to calculate the attention weights, determine which parts of the video should be matched with the text prompts, and output weighted video features.

[0057] In a preferred embodiment, after processing by the expert AdaLN module, a set of prompt words with noise intensity is obtained; by averaging the prompt words in the batch dimension, the operation of compressing multiple prompt words into a single prompt word is achieved; the compressed single prompt word is combined with the noisy image and input into the 3D full attention module to calculate the attention weight; wherein, the expert AdaLN module is used to ensure effective alignment of visual and textual spaces, and seamless transition between different prompt words is achieved by compressing the prompt words that change frame by frame.

[0058] Step S6, denoising queue update and video frame generation: In the denoising queue, only the latent code of the second half is updated to perform a denoising step, and the weighted video features output by the 3D full attention module are used to guide the denoising process of the latent code. After the attention operation, the prompt word is expanded back to the original dimension and input into the feedforward module in the video generation model for further video frame generation processing; after denoising is completed, the latent code at the front of the queue is dequeued and replaced by randomly sampled Gaussian noise at the end of the queue, and the cycle is repeated to generate a video of unlimited length; wherein, the weighted video features output by the 3D full attention module and the latest latent code in the denoising queue, as well as the prompt word expanded back to the original dimension, are combined to generate video frames corresponding to multiple text prompts.

[0059] Step S7, prompt word queue update: When the denoising queue is updated, the prompt word queue is updated synchronously to ensure that each video frame corresponds to the correct text prompt, achieving consistency between video segments and continuity of transition.

[0060] Through the above steps of the present invention, a long video corresponding to multiple text prompts is generated, wherein each video segment retains rich semantic changes and details, and the transition between segments is natural and smooth.

[0061] The following further describes specific embodiments of the present invention, algorithm examples, and experimental verification.

[0062] A fine-tuning-free method for generating long videos with multiple texts utilizes a dual-stream video diffusion model (DVDM) and two innovative modules: frame-by-frame prompts and dimensionally compressed attention. The DVDM improves content quality by processing textual and visual inputs separately via the expert adaptive LayerNorm (AdaLN) module. The frame-by-frame prompt module maintains a queue of prompt words to achieve smooth transitions between video frames, while the dimensionally compressed attention module compresses multiple prompt words to alleviate the training-inference gap and preserve texture details. This method performs well in both automatic metrics and human evaluation, particularly in tracking strength and temporal motion smoothness. This method addresses the challenges of existing models in generating multiple prompt words and long videos, achieving high-quality, temporally consistent, and motion-smooth long videos.

[0063] like Figure 1 As shown, the process begins with a frame-by-frame prompt module, where textual prompts are encoded and organized into prompt queues for frame-by-frame guidance. The initialized diagonal denoising module generates initial video clips based on a two-stream video diffusion model (DVDM), adding noise of varying intensities to each frame before placing them into the denoising queue. In the dimensionally compressed attention module, the prompt words and visual input are first aligned with the corresponding noise intensities. Before applying attention, the prompt words are compressed to remain consistent with the attention span used during pre-training. After the attention operation, the prompt words are expanded back to the original dimension to continue processing through the feedforward module of the video generation model.

[0064] Method initialization

[0065] To initialize the denoising queue, we first generate N frames of video using the base model. We then apply noise of varying intensities to these frames to simulate varying degrees of quality degradation. However, when the total number of frames required, f, is greater than the number of frames generated, N, additional frames are needed to fill the gaps. In this case, the first frame of the video is repeated fN times, and noise of varying intensities is applied to each repeated frame to ensure consistency with the rest of the sequence. The initialized denoising queue can be expressed as:

[0066]

[0067] The latent codes in this queue will not be decoded into frames of the final generated video after denoising. This is because these codes are generated by the base model and may affect the evaluation of experimental results. Therefore, they are only used for initialization.

[0068] FIFO denoising queue update

[0069] Maintain a hint queue with the same length as the denoising queue and update it synchronously when the denoising queue is updated.

[0070] Long videos are generated through a diagonal denoising scheme using a FIFO method. Specifically, a denoising queue is maintained, where the noise level of the latent code is gradually increased to autoregressively generate infinitely long videos. Specifically, for a time step schedule 0 = τ_0 < τ_1 < ... < τ_f = T, its mathematical form is expressed as:

[0071]

[0072] where Φ represents the sampler used in the base model, ∈_θ represents the predicted noise, and the latent code

[0073] z_{τ_i}^{i} represents the potential code of the i-th frame in the denoising queue at the noise stage τ_i.

[0074] Since f is usually greater than N, where N represents the number of video frames that the base model can process at a time, to bridge the gap between training and inference, FIFO maintains the original base model's diffusion step size f unchanged by selecting a fixed-length window of N from the denoising queue every s = N / 2 steps to perform a denoising step. The window can be expressed as:

[0075]

[0076] And in each window, only the latent code of the second half will be updated, so a denoising step can be expressed as:

[0077]

[0078] in, yes is a subset of , and Indicates that in the noise stage τ′ i The i-th frame in the time window is updated. After all windows are updated, an iteration is completed. The latent code at the front of the queue is dequeued after being fully denoised and replaced by randomly sampled Gaussian noise at the end of the queue. This denoising process utilizes the cleaner frames at the front of the queue, ensuring that each latent code is denoised once. This cycle continues, allowing the generation of infinitely long videos.

[0079] Frame-by-frame prompts

[0080] Frame-wise prompting module. The inventors found that the timing of switching the text condition to a new prompt word during the denoising process is crucial. To solve this problem, the present invention proposes a frame-wise prompting method that makes subtle changes to the target in each denoising step to achieve a smooth transition of video generation using different prompt words. The core of frame-wise prompting is the gradual prompt switching timing. Specifically, in order to switch prompt words more smoothly between different frames, the present invention maintains a prompt word queue whose length is the same as the denoising queue, and the two correspond one-to-one.

[0081] Figure 2 The following figure shows the update process of the prompt word queue for f=8, N=4, and n=3. Different colors represent different prompt words, where light yellow and dark yellow represent the same prompt word. Light yellow indicates that its corresponding frame is not included in the final generated video. The dotted box represents a window, while the solid box represents the entire queue. Figure 2 As shown, initially, the prompt word queue is filled with c_0, which is indicated by light yellow, indicating that the corresponding video frame will not be output. At iteration i, the prompt word c_{min{(i-1) / / N,n-1}} is added to the end of the queue and removed from the head of the queue. There are a total of f+n×N iterations, generating n×N frames of video.

[0082] Figure 3 This paper describes how to achieve smooth transition by maintaining a prompt word queue, which shows the interaction between the prompt window and the denoising window when N=4. Different colors represent different prompt words. The vertical axis represents the frame index corresponding to each prompt word, and the horizontal axis represents the noise intensity of the frame. The solid line box represents the prompt window currently input to the model, while the dotted line box represents the prompt window to be input in the next iteration. The prompt word entering the window in the next iteration can be Figure 2 As Figure 3 As shown, the yellow block represents the aligned prompt word embedding c_1, and the blue block represents another prompt word embedding c_2. The coefficient before each prompt word embedding is determined by the number of corresponding color blocks in the solid or dotted box containing the prompt word. For the i-th frame, in the denoising step τ_4′, the present invention uses c_1+3c_2 to guide the denoising process. In the denoising step τ_3′, the present invention uses 2c_1+2c_2 to guide the denoising, which achieves the effect of using different prompt words to denoise the same frame at different time steps. In addition, for the (i+1)-th frame, in the denoising step τ_4′, the present invention uses 2c_1+2c_2 for denoising. This achieves the effect of using different prompt words for different frames at the same time step. Therefore, from the two dimensions of frame and time step, the present invention achieves a smooth transition.

[0083] Dimensionality Compressed Attention

[0084] Dimensionality Compressed Attention Module. To ensure the same attention length as the training phase, the present invention requires a compression mechanism. To this end, the present invention designs a prompt word compression module. Specifically, after the expert AdaLN module, the present invention obtains a set of prompt words with noise intensity, denoted as in yes The method of the present invention is to average these prompt words in the batch dimension, that is:

[0085]

[0086] The compression module compresses N prompt words into a single prompt word. This prompt word is then connected to the noisy image and input into the 3D full attention module. The present invention further emphasizes that after passing through the expert AdaLN module, the visual and textual spaces are effectively aligned. Therefore, compressing the prompt words that change frame by frame is similar to interpolating between prompt words, which enables seamless transitions between different prompt words. In this way, the present invention not only bridges the training-inference gap and retains more details, but also achieves seamless transitions between different prompt words.

[0087] Experimental results

[0088] Figure 4 3 is a comparison chart of experimental results of the embodiment of the present invention and other methods.

[0089] Table 1 shows a quantitative comparison experiment of the present invention in terms of automatic metrics. The performance of the present invention and other methods in the video generation task is evaluated, especially in terms of three automatic metrics: temporal consistency, tracking strength, and temporal motion smoothness.

[0090] Table 1

[0091]

[0092] Table 2

[0093]

[0094] The experimental results in Table 1 show that our method outperforms all other compared methods in terms of automatic metrics, with a particularly strong performance in tracking strength. This demonstrates that our method has a clear advantage in generating high-quality, temporally consistent, and motion-smooth long videos.

[0095] Table 2 is a comparison table of human evaluation. The experimental results show that the present invention has achieved the highest standards in all key indicators of human evaluation, further confirming its significant advantages in generating long videos with high quality, consistent content, smooth transitions, and high alignment with text prompts.

[0096] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0097] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0098] An embodiment of the present invention further provides a processor, which executes a computer program and at least performs the method described above.

[0099] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a magnetic disk memory or a magnetic tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0100] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0101] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0102] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0103] Those skilled in the art will understand that all or part of the steps of the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc. Various media that can store program codes.

[0104] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0105] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0106] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0107] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0108] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art will recognize that, without departing from the scope of the present invention, several equivalent substitutions or obvious variations can be made, and the performance or use of the same should be considered to fall within the scope of protection of the present invention.

Claims

1. A method for generating long multi-text videos without fine-tuning, characterized in that: The following steps are involved: S1. Generate N frames of video using the basic model and add noise of varying intensities to each frame to simulate varying degrees of quality degradation, forming an initial denoising queue. S2. Maintain a prompt word queue, where each prompt word corresponds to a video frame. Initially, the prompt word queue is filled with the first prompt word. S3,processing text and visual inputs through the dual-stream video diffusion model DVDM, which uses two expert adaptive LayerNorm modules to process text and visual modalities respectively, and the time step of the diffusion process is used as a modulation module; S4. During the denoising process, the prompt words are gradually switched to change the prompt words in a smoother way and interpolated in a space closer to the visual domain to achieve a smooth transition of video generation using different prompt words. S5. Compress multiple prompt words into a single representation to alleviate the training-inference gap in DVDM while preserving texture details. Multiple prompt words are compressed into a single prompt word by averaging the prompt words over the batch dimension, which is then concatenated with the noisy image and input into the 3D full attention module to calculate the attention weights, determine which parts of the video should be matched with the text prompt, and output weighted video features. S6. In the denoising queue, only the latent codes of the second half are updated to perform a denoising step. At the same time, the weighted video features output by the 3D full attention module are used to guide the denoising process of the latent codes. After the attention operation, the prompt word is expanded back to the original dimension so that it can be used for the final video generation. After denoising is completed, the latent code at the front of the queue is dequeued and replaced by randomly sampled Gaussian noise at the end of the queue. This cycle is repeated to generate a video of unlimited length. The weighted video features output by the 3D full attention module and the latest latent code in the denoising queue are combined with the prompt word expanded back to the original dimension to generate video frames corresponding to multiple text prompts. S7. While the denoising queue is being updated, the prompt word queue is also updated synchronously so that each video frame corresponds to a correct text prompt.

2. The method for generating a multi-text long video without fine-tuning according to claim 1, wherein: In step S1, based on the N frames of video generated by the basic model, the diagonal denoising module is initialized using DVDM. By initializing the diagonal denoising module, noise of different intensities is added to each frame of video to simulate different degrees of degradation, thereby forming an initial denoising queue.

3. The method for generating a multi-text long video without fine-tuning according to any one of claims 1 to 2, characterized in that: In step S1, initializing the denoising queue specifically includes: When the total number of frames required is greater than the number of video frames generated by the base model at one time, additional frames are generated by reusing the first frame of the video and applying noise of varying intensities to form a complete denoising queue; The potential codes in the denoising queue are used to initialize the denoising process and are not directly used to generate the final video frame after denoising, but are only used as the starting point of the denoising process.

4. The method for generating a multi-text long video without fine-tuning according to any one of claims 1 to 3, characterized in that: Step S4 includes: During the denoising process, the cue words are slightly adjusted in each denoising step to achieve a smooth transition in video generation between different cue words. Maintain a cue word queue that matches the length of the denoising queue and iteratively update the queue to gradually switch the cue words to ensure semantic coherence between video frames. By adjusting the coefficients before the cue words are embedded, different cue words can be applied to the same frame or different frames at different time steps, thereby achieving a smooth transition of video content in both the frame and time step dimensions.

5. The method for generating a multi-text long video without fine-tuning according to claim 4, wherein: In step S4, the frame-by-frame prompting and denoising synchronization specifically include: The initial prompt word is used to fill the queue, and the corresponding video frame is not included in the final output video. During each iteration, according to the number of iterations, a new prompt word is calculated and added to the end of the queue, and the prompt word at the head of the queue is removed to maintain a one-to-one correspondence between the prompt word queue and the denoising queue. In the denoising step, the coefficients before the cue word embedding are determined based on the position of the cue word in the current and future iterations; among them, distinguishing the cue window to be input into the model in the current and next iterations ensures the correct application of the cue word and the smooth transition of the video content.

6. The method for generating a multi-text long video without fine-tuning according to any one of claims 1 to 5, characterized in that: The dimensionally compressed attention processing in step S5 further includes: After processing by the expert AdaLN module, a set of prompt words with noise intensity is obtained; Compressing multiple prompt words into a single prompt word by averaging the prompt words in the batch dimension; The compressed single prompt word is combined with the noisy image and input into the 3D full attention module to calculate the attention weight; Among them, the expert AdaLN module is used to ensure the effective alignment of visual and textual spaces, and seamless transition between different prompt words is achieved by compressing the prompt words that change frame by frame.

7. The method for generating a multi-text long video without fine-tuning according to any one of claims 1 to 6, characterized in that: In step S6, after processing by the 3D full attention module, the prompt word is expanded back to the original dimension and input into the feedforward module of the video generation model for further video frame generation processing.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for generating a multi-text long video without fine-tuning as described in any one of claims 1 to 7 is implemented.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for generating a multi-text long video without fine-tuning as described in any one of claims 1 to 7 is implemented.

Images