Multi-text long video generation method without fine tuning

Abstract

The invention discloses a multi-text long video generation method without fine tuning, which is called DuFree, a double-stream video diffusion model (DVDM) is utilized, and a frame-by-frame prompt and dimension compression attention module is designed, so that smooth transition among a plurality of prompt words is ensured, and meanwhile, details are reserved. And the DVDM respectively processes the text and visual input through an expert adaptive LayerNorm (AdaLN) module, so that the content quality is improved. The frame-by-frame prompt module realizes smooth transition between video frames by maintaining a prompt word queue; the dimension compression attention module compresses a plurality of cue words, relieves training-reasoning gaps and retains texture details. The method is excellent in performance in automatic measurement and human evaluation, especially in the aspects of tracking intensity and time motion smoothness, challenges of an existing model in multi-cue word and long video generation are solved, and high-quality, time-consistent and smooth-motion long video generation is achieved.

Description

Technical Field

[0001] The invention relates to video generation technology, and in particular to a method for generating multi-text long videos without fine-tuning. Background Art

[0002] The rapid development of diffusion models has made significant progress in generation tasks, especially in the field of vision. The diffusion model, originally used for image synthesis, is able to generate highly detailed images from text descriptions through a cross-attention mechanism, demonstrating its strong ability in spatial content modeling.

[0003] Building on the success of image generation, people naturally extend their interest to more complex fields such as videos, where temporal dynamics become crucial. Therefore, many studies have been devoted to applying diffusion-based methods to video generation, trying to exploit its ability to process time-series frames. Recently, a new two-stream diffusion model architecture has been proposed, which emphasizes that better alignment between text and visual content can significantly improve performance. The present invention adopts this new architectural form, namely the two-stream video diffusion model (DVDM). To achieve this, the model uses two expert adaptive LayerNorm modules to handle the two modalities respectively, and the time step of the diffusion process is used as a modulation module, which significantly improves the quality of the generated content.

[0004] Despite these advances, current diffusion models for video generation still face significant challenges. Compared to images, videos are high-dimensional data, requiring larger models and more extensive training datasets. As a result, existing methods are often trained on datasets containing only a small number of video frames, limiting their ability to generate long videos. In addition, these methods perform poorly when generating videos containing multiple scenes or complex actions. This limitation stems from the challenges encountered when training models with multiple cues, especially when video datasets containing rich multi-stage narratives are scarce.

[0005] A straightforward approach to addressing the above issues without additional training is to use the base model to generate separate video clips for each cue word and then stitch them together. This approach can create long videos containing multiple scenes with rich semantic changes. However, this often results in poor consistency between clips and discontinuous transitions. Diffusion models are very sensitive to changes in cue words, and even small changes in text cues can lead to significant changes in the content of the generated video. To address this problem, FreeNoise uses cue word interpolation and injects them within a specific range of denoising steps to achieve video clip transitions. However, its effectiveness is limited because different frames require switching cue words at different denoising steps to achieve the best results, and the interpolation of cue word embeddings does not always produce semantically coherent transitions.

[0006] It should be noted that the information disclosed in the above background technology section is only used for understanding the background of the present application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention

[0007] The main purpose of the present invention is to overcome the defects existing in the above-mentioned background technology and provide a method for generating multi-text long videos without fine-tuning.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] A method for generating multi-text long videos without fine-tuning, comprising the following steps:

[0010] S1. Generate N frames of video using the basic model and add noise of different intensities to each frame to simulate different degrees of degradation to form an initial denoising queue;

[0011] S2, maintaining a prompt word queue, each prompt word corresponds to a video frame, and the prompt word queue is initially filled with the first prompt word;

[0012] S3, processing text and visual inputs through a dual-stream video diffusion model DVDM, which uses two expert adaptive LayerNorm modules to process text and visual modalities respectively, and the time step of the diffusion process is used as a modulation module;

[0013] S4, during the denoising process, gradually switch the prompt words, change the prompt words in a smoother way, and interpolate in a space closer to the visual domain to achieve a smooth transition of video generation using different prompt words;

[0014] S5. Compress multiple cue words into one representation to alleviate the training-inference gap in DVDM while preserving texture details; wherein, multiple cue words are compressed into a single cue word by averaging the cue words over the batch dimension, which is then concatenated with the noisy image and input into the 3D full attention module to calculate the attention weights, determine which parts of the video should be matched with the text cue, and output weighted video features;

[0015] S6. In the denoising queue, only the latent code of the second half is updated to perform a denoising step, and the weighted video features output by the 3D full attention module are used to guide the denoising process of the latent code. After the attention operation, the prompt word is expanded back to the original dimension so that it can be used for the final video generation. After the denoising is completed, the latent code at the front of the queue is dequeued and replaced by randomly sampled Gaussian noise at the end of the queue, and the cycle is repeated to generate a video of unlimited length. The weighted video features output by the 3D full attention module and the latest latent code in the denoising queue are combined with the prompt word expanded back to the original dimension to generate video frames corresponding to multiple text prompts.

[0016] S7. While the denoising queue is being updated, the prompt word queue is also being updated synchronously, so that each video frame corresponds to a correct text prompt.

[0017] Furthermore, in step S1, based on the basic model generating N frames of video, the diagonal denoising module is initialized using DVDM, and noise of different intensities is added to each frame of video by initializing the diagonal denoising module to simulate different degrees of degradation, thereby forming an initial denoising queue.

[0018] Furthermore, in step S1, initializing the denoising queue specifically includes:

[0019] When the total number of frames required is greater than the number of video frames generated by the base model at one time, additional frames are generated by reusing the first frame of the video and applying noise of varying intensities to form a complete denoising queue;

[0020] The potential codes in the denoising queue are used to initialize the denoising process and are not directly used to generate the final video frame after denoising, but are only used as the starting point of the denoising process.

[0021] Further, step S4 includes:

[0022] In the denoising process, the cue words are slightly adjusted in each denoising step to achieve a smooth transition of video generation between different cue words;

[0023] Maintain a prompt word queue that matches the length of the denoising queue, and iteratively update the queue to switch prompt words in a progressive manner to ensure semantic coherence between video frames;

[0024] By adjusting the coefficients before the cue words are embedded, different cue words can be applied to the same frame or different frames at different time steps, thereby achieving a smooth transition of video content in two dimensions: frame and time step.

[0025] Furthermore, in step S4, the frame-by-frame prompting and denoising synchronization specifically include:

[0026] The initial prompt word is used to fill the queue, and the corresponding video frame is not included in the final output video. In each iteration, according to the number of iterations, a new prompt word is calculated and added to the end of the queue, and the prompt word at the head of the queue is removed to maintain a one-to-one correspondence between the prompt word queue and the denoising queue.

[0027] In the denoising step, the coefficients before the cue word embedding are determined based on the position of the cue word in the current and future iterations; among them, the cue window to be input into the model in the current and next iterations is distinguished to ensure the correct application of the cue word and the smooth transition of the video content.

[0028] Furthermore, the dimension compression attention processing in step S5 further includes:

[0029] After being processed by the expert AdaLN module, a set of prompt words with noise intensity is obtained;

[0030] By averaging the prompt words in a batch dimension, the operation of compressing multiple prompt words into a single prompt word is realized;

[0031] The compressed single prompt word is combined with the noisy image and input into the 3D full attention module to calculate the attention weight;

[0032] Among them, the expert AdaLN module is used to ensure the effective alignment of visual and textual spaces, and seamless transition between different prompt words is achieved by compressing the prompt words that change frame by frame.

[0033] Furthermore, in step S6, after being processed by the 3D full attention module, the prompt word is expanded back to the original dimension and input into the feedforward module of the video generation model for further video frame generation processing.

[0034] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the method for generating a multi-text long video without fine-tuning.

[0035] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the method for generating a multi-text long video without fine-tuning is implemented.

[0036] The present invention has the following beneficial effects:

[0037] This paper provides a method for generating long multi-text videos without fine-tuning, called DuFree, and proposes the first DVDM-based framework for generating long multi-cue words without fine-tuning. The paper innovatively designs frame-by-frame cues and dimensionally compressed attention, combined with a two-stream video diffusion model (DVDM), to achieve multi-text long video generation without fine-tuning, ensuring smooth transitions between multiple cues while preserving details. Through DVDM, the paper uses two expert adaptive LayerNorm (AdaLN) modules to process text and visual modalities respectively, and uses the time step of the diffusion process as a modulation module to significantly improve the quality of generated content. The frame-by-frame cues module achieves smooth transitions between different frames by maintaining a cues queue that matches the length of the denoising queue, while the dimensionally compressed attention module alleviates the gap between training and reasoning by compressing multiple cues into a single representation, while preserving important texture details. Experiments show that compared with other methods, DuFree performs better in terms of segment temporal consistency, tracking strength, and temporal motion smoothness, achieving state-of-the-art results and demonstrating excellent performance. The present invention achieves smooth transitions between video clips by switching prompt words in a timely manner during the denoising process, solving the challenges of existing video generation diffusion models in processing multiple prompt words and generating long videos, especially when data sets are scarce. The present invention verifies the effectiveness of the method through experiments, demonstrating its significant advantages in generating high-quality, time-consistent, and motion-smooth long videos.

[0038] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is an overview of the DuFree framework of the multi-text long video generation method according to an embodiment of the present invention.

[0040] Figure 2 It is a schematic diagram of updating the prompt queue in the frame-by-frame prompt scheme of an embodiment of the present invention.

[0041] Figure 3 It is a schematic diagram of how the frame-by-frame prompting solution of an embodiment of the present invention takes effect.

[0042] Figure 4 It is a comparison chart of experimental results of the embodiment of the present invention and other methods. DETAILED DESCRIPTION

[0043] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.

[0044] It should be noted that when an element is referred to as being "fixed to" or "disposed on" another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element. In addition, connection can be used for fixing as well as for coupling or communication.

[0045] It should be understood that the orientation or position relationship indicated by terms such as "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside" and "outside" are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0046] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0047] The present invention proposes a method for generating long videos with multiple cues without adjustment, called DuFree, which aims to generate longer videos without affecting the quality by using the advanced two-stream video diffusion model (DVDM) for reasoning. The core idea is to gradually switch the cues during the denoising process, change the cues in a smoother way, and interpolate in a space closer to the visual domain. The present invention specifically includes two important innovations: frame-by-frame cues and dimensionally compressed attention. The core is to switch the cues earlier in the denoising process, which is applicable to the frames that appear later in each video clip, thereby increasing the number of denoising steps affected by subsequent cues, and introducing a novel cues interpolation method to achieve a smooth transition. The present invention achieves this through two specific modules: frame-by-frame cues and dimensionally compressed attention. Frame-by-frame cues maintain a cues queue, each cues corresponding to a video frame. By projecting the cues embedding into the visual alignment space using the adaptive LayerNorm of DVDM, the present invention interpolates to obtain better visual transition values, thereby helping to guide the generation of smooth video transitions. The dimension-compressed attention module compresses multiple cue words into one representation, alleviating the training-inference gap in DVDM when generating long videos containing multiple cue words while preserving texture details.

[0048] See also Figure 1The embodiment of the present invention provides a method for generating a multi-text long video without fine-tuning, comprising the following steps:

[0049] Step S1, initializing the denoising queue (video queue with noise): using the basic model to generate N frames of video, and adding noise of different intensities to each frame to simulate different degrees of degradation, forming an initial denoising queue.

[0050] In a preferred embodiment, specifically, based on the generation of N frames of video by the basic model, the characteristics of the dual-stream video diffusion model (DVDM) are used to initialize the diagonal denoising module, and different intensities of noise are added to each frame of video by initializing the diagonal denoising module to simulate different degrees of degradation, thereby forming an initial denoising queue to prepare for the subsequent denoising process. Initializing the denoising queue specifically includes: when the total number of frames required is greater than the number of video frames generated by the basic model at one time, additional frames are generated by reusing the first frame of the video and applying noise of different intensities to form a complete denoising queue; the latent code in the denoising queue is used to initialize the denoising process, and is not directly used to generate the final video frame after denoising, but only as the starting point of the denoising process.

[0051] Step S2, constructing a frame-by-frame prompt queue: maintaining a prompt word queue, each prompt word corresponds to a video frame, and the prompt word queue is initially filled with the first prompt word;

[0052] Step S3, dual-stream video diffusion model processing: using the dual-stream video diffusion model DVDM to process text and visual inputs, wherein the model uses two expert adaptive LayerNorm (AdaLN) modules to process text and visual modalities respectively, and the time step of the diffusion process is used as a modulation module to improve the quality of generated content;

[0053] Step S4, frame-by-frame prompting and denoising synchronization: During the denoising process, the prompt words are gradually switched, the prompt words are changed in a smoother way, and interpolation is performed in a space closer to the visual domain to achieve a smooth transition of video generation using different prompt words.

[0054] In a preferred embodiment, during the denoising process, a smooth transition of video generation between different prompt words is achieved by making slight adjustments to the prompt words in each denoising step; a prompt word queue matching the length of the denoising queue is maintained, and the queue is iteratively updated to switch the prompt words in a progressive manner to ensure semantic coherence between video frames; and different prompt words are applied to the same frame or different frames at different time steps by adjusting the coefficients before the prompt words are embedded, thereby achieving a smooth transition of video content in both the frame and time step dimensions.

[0055] In a preferred embodiment, the synchronization of frame-by-frame prompting and denoising specifically includes: initializing a prompt word queue, wherein the initial prompt word c0 is used to fill the queue, and its corresponding video frame is not included in the final output video; in each iteration, according to the number of iterations, a new prompt word is calculated and added to the end of the queue, and the prompt word at the head of the queue is removed to maintain a one-to-one correspondence between the prompt word queue and the denoising queue; in the denoising step, the coefficient of the prompt word before embedding is determined based on the position of the prompt word in the current and future iterations; wherein, the prompt window to be input into the model in the current and next iterations is distinguished to ensure the correct application of the prompt word and the smooth transition of the video content.

[0056] Step S5, dimensionally compressed attention processing: compress multiple cue words into one representation to alleviate the training-inference gap in DVDM while retaining texture details; wherein, by averaging the cue words over the batch dimension, N cue words are compressed into a single cue word, which is then concatenated with the noisy image and input into the 3D full attention module to calculate the attention weights, determine which parts of the video should be matched with the text cue, and output weighted video features.

[0057] In a preferred embodiment, after processing by the expert AdaLN module, a group of prompt words with noise intensity are obtained; by averaging the prompt words in the batch dimension, the operation of compressing multiple prompt words into a single prompt word is realized; the compressed single prompt word is combined with the noisy image and input into the 3D full attention module to calculate the attention weight; wherein, the expert AdaLN module is used to ensure the effective alignment of visual and textual spaces, and the seamless transition between different prompt words is achieved by compressing the prompt words that change frame by frame.

[0058] Step S6, denoising queue update and video frame generation: In the denoising queue, only the latent code of the second half is updated to perform a denoising step, and the weighted video features output by the 3D full attention module are used to guide the denoising process of the latent code. After the attention operation, the prompt word is expanded back to the original dimension and input into the feedforward module in the video generation model for further video frame generation processing; after denoising is completed, the latent code in front of the queue is dequeued and replaced by randomly sampled Gaussian noise at the end of the queue, and the cycle is repeated to generate a video of unlimited length; wherein, the weighted video features output by the 3D full attention module and the latest latent code in the denoising queue, as well as the prompt word expanded back to the original dimension, are combined to generate video frames corresponding to multiple text prompts.

[0059] Step S7, updating the prompt word queue: while updating the denoising queue, the prompt word queue is also updated synchronously to ensure that each video frame corresponds to the correct text prompt, thus achieving consistency between video clips and continuity of transition.

[0060] Through the above steps of the present invention, a long video corresponding to multiple text prompts is generated, wherein each video segment retains rich semantic changes and details, and the transition between segments is natural and smooth.

[0061] The following further describes specific embodiments of the present invention, algorithm examples and experimental verification.

[0062] A fine-tuning-free method for multi-text long video generation uses a dual-stream video diffusion model (DVDM) and two innovative modules: frame-by-frame prompting and dimensionally compressed attention. DVDM processes text and visual inputs separately through the expert adaptive LayerNorm (AdaLN) module to improve content quality. The frame-by-frame prompting module achieves smooth transitions between video frames by maintaining a queue of prompt words; the dimensionally compressed attention module compresses multiple prompt words to alleviate the training-inference gap and preserve texture details. This method performs well in both automatic metrics and human evaluations, especially in tracking strength and temporal motion smoothness. It addresses the challenges of existing models in multi-prompt word and long video generation and achieves high-quality, temporally consistent and motion-smooth long video generation.

[0063] like Figure 1 As shown, the process starts with a frame-by-frame prompt module, where textual prompts are encoded and organized into prompt queues for frame-by-frame guidance. The initialized diagonal denoising module generates initial video clips based on a two-stream video diffusion model (DVDM), adding noise of different intensities to each frame before putting them into the denoising queue. In the dimensionally compressed attention module, the prompt words and visual inputs are first aligned with the corresponding noise intensities. Before applying attention, the prompt words are compressed to keep consistent with the attention span used during pre-training. After the attention operation, the prompt words are expanded back to the original dimension to continue processing through the feed-forward module of the video generation model.

[0064] Method Initialization

[0065] To initialize the denoising queue, we first generate N frames of video using the base model. Then we apply noise of different strengths to these frames to simulate different degrees of degradation. However, when the total number of frames f required is greater than the number of frames N generated, additional frames are needed to fill the gap. In this case, the first frame of the video is repeated fN times, and noise of different strengths is applied to each repeated frame to ensure consistency with other sequences. The initialized denoising queue can be expressed as:

[0066]

[0067] The latent codes in this queue will not be decoded into frames of the final generated video after denoising. This is because these codes are generated by the base model and may affect the evaluation of experimental results. Therefore, they are only used as initialization.

[0068] FIFO denoising queue update

[0069] Maintain a hint queue with the same length as the denoising queue and update it synchronously when the denoising queue is updated.

[0070] Long videos are generated by a diagonal denoising scheme of the FIFO method. Specifically, a denoising queue is maintained, in which the noise level of the potential code is gradually increased to autoregressively generate infinitely long videos. Specifically, for the time step schedule 0 = τ_0 < τ_1 < ... < τ_f = T, its mathematical form is expressed as:

[0071]

[0072] where Φ represents the sampler used in the base model, ∈_θ represents the predicted noise, and the latent code

[0073] z_{τ_i}^{i} represents the potential code of the i-th frame in the denoising queue at the noise stage τ_i.

[0074] Since f is usually greater than N, where N represents the number of video frames that the base model can process at one time. In order to narrow the gap between training and inference, FIFO keeps the diffusion step size f of the original base model unchanged by selecting a fixed-length window of N from the denoising queue every s=N / 2 steps to perform a denoising step. The window can be expressed as:

[0075]

[0076] And in each window, only the latent code of the second half will be updated, so a denoising step can be expressed as:

[0077]

[0078] in, yes is a subset of , and Indicates that in the noise stage τ′ i The i-th frame in the time window is taken. After all windows are updated, one iteration is completed. The latent code at the front of the queue is dequeued after being fully denoised and replaced by randomly sampled Gaussian noise at the end of the queue. This denoising process utilizes the cleaner frames at the front of the queue, ensuring that each latent code is denoised once. This cycle continues, allowing the generation of infinitely long videos.

[0079] Frame by frame prompts

[0080] Frame-wise prompting module. The inventors found that the timing of switching text conditions to new prompt words during the denoising process is crucial. In order to solve this problem, the present invention proposes a frame-wise prompting method, which makes subtle changes to the target in each denoising step to achieve a smooth transition of video generation using different prompt words. The core of frame-wise prompting is the progressive prompt switching timing. Specifically, in order to switch prompt words more smoothly between different frames, the present invention maintains a prompt word queue whose length is the same as the denoising queue, and the two correspond one to one.

[0081] Figure 2 FIG. 1 shows an explanation of the update process of the prompt word queue for f=8, N=4, and n=3. Different colors represent different prompt words, where light yellow and dark yellow represent the same prompt word. Light yellow indicates that its corresponding frame is not included in the final generated video. The dotted box represents a window, while the solid box represents the entire queue. Figure 2 As shown, initially, the prompt word queue is filled with c_0, which is indicated by light yellow in the figure, indicating that the corresponding video frame will not be output. At the i-th iteration, the prompt word c_{min{(i-1) / / N,n-1}} is added to the end of the queue and the head of the queue is removed. There are a total of f+n×N iterations, generating n×N frames of video.

[0082] Figure 3 Describes how to achieve smooth transition by maintaining a prompt word queue, showing the interaction between the prompt window and the denoising window when N=4. Different colors represent different prompt words. The vertical axis represents the frame index corresponding to each prompt word, and the horizontal axis represents the noise intensity of the frame. The solid line box represents the prompt window currently input to the model, while the dotted line box represents the prompt window to be input in the next iteration. The prompt word entering the window in the next iteration can be Figure 2 As Figure 3 As shown, the yellow block is represented as an aligned cue word embedding c_1, and the blue block represents another cue word embedding c_2. The coefficient before each cue word embedding is determined by the number of corresponding color blocks in the solid or dotted box containing the cue word. For the i-th frame, in the denoising step τ_4′, the present invention uses c_1+3c_2 to guide the denoising process. In the denoising step τ_3′, the present invention uses 2c_1+2c_2 to guide the denoising, which achieves the effect of using different cue words to denoise the same frame at different time steps. In addition, for the (i+1)th frame, in the denoising step τ_4′, the present invention uses 2c_1+2c_2 for denoising. This achieves the effect of using different cue words for different frames at the same time step. Therefore, from the two dimensions of frame and time step, the present invention achieves a smooth transition.

[0083] Dimensionality Compressed Attention

[0084] Dimensionality Compressed Attention Module. In order to ensure the same attention length as the training phase, the present invention requires a compression mechanism. To this end, the present invention designs a prompt word compression module. Specifically, after the expert AdaLN module, the present invention obtains a set of prompt words with noise intensity, denoted as in yes The method of the present invention is to average these prompt words in the batch dimension, that is:

[0085]

[0086] The compression module compresses N prompt words into a single prompt word. This prompt word is then connected to the noisy image and input into the 3D full attention module. The present invention further emphasizes that after passing through the expert AdaLN module, the visual and textual spaces are effectively aligned. Therefore, compressing the prompt words that change frame by frame is similar to interpolating between prompt words, which enables seamless transitions between different prompt words. In this way, the present invention not only bridges the training-inference gap and retains more details, but also achieves seamless transitions between different prompt words.

[0087] Experimental Results

[0088] Figure 4 It is a comparison chart of experimental results of the embodiment of the present invention and other methods.

[0089] Table 1 shows the quantitative comparison experiment of the present invention in terms of automatic metric. The performance of the present invention and other methods in the video generation task is evaluated, especially in terms of the three automatic metric indicators of temporal consistency, tracking strength and temporal motion smoothness.

[0090] Table 1

[0091]

[0092] Table 2

[0093]

[0094] The experimental results in Table 1 show that the present invention outperforms other compared methods in terms of automatic metric indicators, especially in terms of tracking strength. This shows that the present invention has obvious advantages in generating high-quality, time-consistent, and smooth motion long videos.

[0095] Table 2 is a human evaluation comparison table. The experimental results show that the present invention has reached the highest standards in the key indicators of human evaluation, further confirming its significant advantages in generating long videos with high quality, consistent content, smooth transitions and high alignment with text prompts.

[0096] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0097] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0098] An embodiment of the present invention further provides a processor, wherein the processor executes a computer program and at least executes the method described above.

[0099] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0100] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0101] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0102] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0103] Those skilled in the art can understand that: all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0104] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0105] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0106] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0107] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0108] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A method for generating long multi-text videos without fine-tuning, characterized in that: The following steps are involved: S1. Generate N frames of video using the basic model and add noise of different intensities to each frame to simulate different degrees of degradation to form an initial denoising queue; S2, maintaining a prompt word queue, each prompt word corresponds to a video frame, and the prompt word queue is initially filled with the first prompt word; S3, processing text and visual inputs through a dual-stream video diffusion model DVDM, which uses two expert adaptive LayerNorm modules to process text and visual modalities respectively, and the time step of the diffusion process is used as a modulation module; S4, during the denoising process, gradually switch the prompt words, change the prompt words in a smoother way, and interpolate in a space closer to the visual domain to achieve a smooth transition of video generation using different prompt words; S5. Compress multiple cue words into one representation to alleviate the training-inference gap in DVDM while preserving texture details; wherein, multiple cue words are compressed into a single cue word by averaging the cue words over the batch dimension, which is then concatenated with the noisy image and input into the 3D full attention module to calculate the attention weights, determine which parts of the video should be matched with the text cue, and output weighted video features; S6. In the denoising queue, only the latent code of the second half is updated to perform a denoising step, and the weighted video features output by the 3D full attention module are used to guide the denoising process of the latent code. After the attention operation, the prompt word is expanded back to the original dimension so that it can be used for the final video generation. After the denoising is completed, the latent code at the front of the queue is dequeued and replaced by randomly sampled Gaussian noise at the end of the queue, and the cycle is repeated to generate a video of unlimited length. The weighted video features output by the 3D full attention module and the latest latent code in the denoising queue are combined with the prompt word expanded back to the original dimension to generate video frames corresponding to multiple text prompts. S7. While the denoising queue is being updated, the prompt word queue is also being updated synchronously, so that each video frame corresponds to a correct text prompt.

2. The method for generating a multi-text long video without fine-tuning as claimed in claim 1, characterized in that: In step S1, based on the basic model generating N frames of video, the diagonal denoising module is initialized using DVDM. By initializing the diagonal denoising module, noise of different intensities is added to each frame of video to simulate different degrees of degradation, thereby forming an initial denoising queue.

3. The method for generating a multi-text long video without fine-tuning according to any one of claims 1 to 2, characterized in that: In step S1, initializing the denoising queue specifically includes: When the total number of frames required is greater than the number of video frames generated by the base model at one time, additional frames are generated by reusing the first frame of the video and applying noise of varying intensities to form a complete denoising queue; The potential codes in the denoising queue are used to initialize the denoising process and are not directly used to generate the final video frame after denoising, but are only used as the starting point of the denoising process.

4. The method for generating a multi-text long video without fine-tuning according to any one of claims 1 to 3, characterized in that: Step S4 includes: In the denoising process, the cue words are slightly adjusted in each denoising step to achieve a smooth transition of video generation between different cue words; Maintain a prompt word queue that matches the length of the denoising queue, and iteratively update the queue to switch prompt words in a progressive manner to ensure semantic coherence between video frames; By adjusting the coefficients before the cue words are embedded, different cue words can be applied to the same frame or different frames at different time steps, thereby achieving a smooth transition of video content in two dimensions: frame and time step.

5. The method for generating a multi-text long video without fine-tuning as claimed in claim 4, characterized in that: In step S4, the frame-by-frame prompting and denoising synchronization specifically include: The initial prompt word is used to fill the queue, and the corresponding video frame is not included in the final output video. In each iteration, according to the number of iterations, a new prompt word is calculated and added to the end of the queue, and the prompt word at the head of the queue is removed to maintain a one-to-one correspondence between the prompt word queue and the denoising queue. In the denoising step, the coefficients before the cue word embedding are determined based on the position of the cue word in the current and future iterations; among them, the cue window to be input into the model in the current and next iterations is distinguished to ensure the correct application of the cue word and the smooth transition of the video content.

6. The method for generating a multi-text long video without fine-tuning according to any one of claims 1 to 5, characterized in that: The dimension compression attention processing in step S5 further includes: After being processed by the expert AdaLN module, a set of prompt words with noise intensity is obtained; By averaging the prompt words in a batch dimension, the operation of compressing multiple prompt words into a single prompt word is realized; The compressed single prompt word is combined with the noisy image and input into the 3D full attention module to calculate the attention weight; Among them, the expert AdaLN module is used to ensure the effective alignment of visual and textual spaces, and seamless transition between different prompt words is achieved by compressing the prompt words that change frame by frame.

7. The method for generating a multi-text long video without fine-tuning according to any one of claims 1 to 6, characterized in that: In step S6, after being processed by the 3D full attention module, the prompt word is expanded back to the original dimension and input into the feedforward module of the video generation model for further video frame generation processing.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for generating a multi-text long video without fine-tuning as described in any one of claims 1 to 7 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for generating a multi-text long video without fine-tuning as described in any one of claims 1 to 7 is implemented.

Images