Lightweight text video generation method and device

Abstract

The invention discloses a lightweight text video generation method and device, the method combines cue word learning and time sequence perception DDIM inversion technology, the editing flexibility is improved by introducing a shared cue word token, meanwhile, the cross-frame consistency is enhanced through a non-shared frame specific token, and meanwhile, the text video generation efficiency is improved. The DDIM inversion of time sequence perception of bilateral filtering is adopted to further optimize the coherence between the video frames, and the text generation image video editing performance based on the diffusion algorithm is improved with the lowest training cost. According to the method, the time sequence consistency, the semantic alignment, the inter-frame consistency and the video quality in a text generation video generation framework are improved under the condition that extra training is not needed.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and more specifically, to a lightweight text-to-video generation method and device. Background Art

[0002] The rapid development of text-to-image techniques based on diffusion models has paved the way for low-cost video editing applications. These methods are built on pre-trained models and provide an efficient alternative to training resource-intensive video-specific models. However, due to the frame-independence of text-to-image techniques, achieving temporal consistency in these applications remains a challenge. Existing methods address this issue by fine-tuning the temporal layers or propagating temporal features during the inference phase, but these methods often require significant training costs or have difficulty generating high-quality and temporally coherent video outputs.

[0003] Text-to-Video (T2V) is a rapidly developing area in the field of artificial intelligence, which aims to transform textual prompts into realistic videos. In recent years, progress has led to significant progress in T2V generation and editing, especially in the field of diffusion models. Among them, Diffusion Transformer (DiT), as a leading architecture, generates high-resolution and high-quality video output through a full spatiotemporal attention mechanism. However, the computational cost of generating even a few seconds of video is extremely high, which poses a huge challenge to its widespread application in low-resource environments.

[0004] In contrast, methods based on pre-trained text-to-image (T2I) diffusion models have attracted much attention due to their lower computational requirements. For example, methods such as Animatediff, Tune-A-Video, and TokenFlow, although they fail to reach the quality of dedicated T2V models, provide more cost-effective solutions for many application scenarios. There are two main strategies in this field: training-based methods and training-free methods.

[0005] Training-based methods initially focused on directly fine-tuning the temporal or attention layers to enforce temporal consistency and improve video editing performance. While effective, this approach often requires high training costs and has limited scalability. To address these issues, adapter-based training methods have been introduced, which provide a more efficient alternative to reduce training costs and enhance generalization capabilities by integrating adapters in the temporal or attention layers. However, most adapters still have relatively large parameter sizes and their generalization capabilities are also limited.

[0006] Training-free methods such as TokenFlow achieve temporal consistency by propagating features between adjacent frames during the inference phase. Although this approach reduces training costs, it usually leads to lower quality output.

[0007] Text-to-image (T2I) techniques have made significant progress in the field of artificial intelligence through advanced methods such as generative adversarial networks (GANs) and diffusion models, but extending these techniques to the task of text-to-video (T2V) remains challenging. Current T2V methods include inversion and sampling methods that optimize the diffusion process, upstream methods that simplify fine-tuning through adapters, and downstream methods that use complex attention mechanisms to improve temporal and semantic consistency. For example, methods such as Gen-L-Video, FLATTEN, and StableVideo address the temporal consistency problem when generating long videos, while methods such as ControlVideo and MagicProp focus on improving the quality of single-frame videos. However, existing 2D UNet-based T2V models usually need to be trained from scratch, which limits the flexibility of pre-trained T2I models and relies on complex temporal layers, resulting in high computational cost and poor consistency. These methods lack a unified, efficient, and lightweight training process, making it difficult to achieve strong generalization capabilities and plug-and-play ease of use.

[0008] Another key challenge in video editing is the inversion process, which is usually required to generate the edited output. Traditional methods such as frame-independent DDIM inversion usually lack optimization for videos, resulting in inconsistency between frames. Denoising Diffusion Implicit Model (DDIM) inversion takes advantage of its reversibility to precisely control the content in the latent space without regenerating the entire image. This technique performs well in image editing and can fine-tune the shape, style, and details of objects while maintaining consistency. Some enhancement techniques such as EasyInv and ReNoise optimize the results by iteratively adding and denoising, while Eta Inversion introduces a time- and region-dependent η function to improve editing diversity. In addition, MasaCtrl converts images into noise representations to recognize object layouts, and Portrait Diffusion achieves efficient image blending by merging Q, K, and V values. Although existing DDIM inversion techniques mainly focus on image editing, there is currently no DDIM inversion strategy optimized for video generation.

[0009] Therefore, how to ensure the temporal consistency, semantic alignment, and inter-frame consistency in the text-to-video (T2V) generation framework without additional training, so as to output high-quality videos, has become one of the urgent problems to be solved. Summary of the invention

[0010] The object of the present invention is to provide a lightweight text-to-video method and device, which improves the temporal consistency, semantic alignment, inter-frame consistency and video quality in a text-to-video (T2V) generation framework without the need for additional training.

[0011] In order to achieve the above object, the present invention adopts the following technical solutions:

[0012] A lightweight method for generating videos from text, which takes text as input and applies it to a diffusion model of a 2D U-Net architecture, comprising the following steps:

[0013] S1: U-Net timing adaptation step, integrating a lightweight adapter module in U-Net for timing fine-tuning;

[0014] S2: semantic alignment adaptation step, which combines shared token embedding of global context with dynamic non-shared token embedding for frame-specific information to achieve semantic alignment;

[0015] S3: DDIM video inversion step, which integrates bilateral filtering that considers both spatial proximity and intensity similarity into the DDIM framework to achieve temporally coherent video output.

[0016] Preferably, the step S1 specifically includes:

[0017] S11: U-Net image reconstruction step, using the U-Net model to predict noise in each denoising step to gradually reconstruct a clear image.

[0018] x t+1 =x t +∈ t -θ(x t ,t)

[0019] Among them, x t+1 represents the image of step t, ∈ t To predict noise, θ is the U-Net model;

[0020] S12: Intermediate feature map extraction step, a hook is registered for each time step to extract the intermediate feature map:

[0021]

[0022] Among them, x is the input feature, B l,b With A l,b is a learnable lightweight low-rank matrix parameter;

[0023] S13: Temporal consistency loss characterization step, timing loss function Reduce the inconsistency of feature representation by minimizing the change in similarity between frames:

[0024]

[0025] The similarity function of adjacent feature maps is:

[0026]

[0027] where F t and F t+1 represents the feature maps extracted at consecutive time steps t and t+1 respectively;

[0028] S14: Diffusion loss characterization step, the standard diffusion loss is defined as follows:

[0029]

[0030] S15: Objective function generation step, the objective function combines the temporal consistency loss and the diffusion loss:

[0031]

[0032] where λ temporal and λ diffusion They are set to 1 and 0.01 respectively to balance the contribution of each loss term.

[0033] Preferably, the step S2 specifically includes:

[0034] S21: Visual hidden feature extraction step. For the input image I, the visual encoder of the CLIP model extracts the visual hidden feature H vision ∈R B×N×d , where B is the batch size, N = 50 is the sequence length, and d = 768 is the feature dimension;

[0035] S22: Non-shared subset generation step, averaging every two adjacent feature vectors on the second dimension to generate a non-shared subset H sub ∈R B×N / 2×d , whose expression is:

[0036]

[0037] for Apply the projection matrix W∈R d×d To H sub , to maintain the feature dimension:

[0038] Z unshare =H sub W,W = nn.Linear(d,d);

[0039] S23: Text embedding step. Text embedding is used for time-series-aware fine-tuning and is constructed as:

[0040]

[0041] Among them, T share is a shared token embed, From the normal distribution Initialization, Zframe is a frame-specific non-shared token, is the concatenation operation of conditional and unconditional embedding in the first dimension of the sequence, and [·] represents concatenation in the sequence dimension.

[0042] Preferably, the step S2 further comprises:

[0043] S24: Training update step, during the training process, the parameters (θ) of the CLIP visual encoder are kept frozen, and only the adapter parameters, the projection layer (φ) of the shared embedding and the non-shared tokens are iteratively optimized to minimize the following loss function:

[0044]

[0045] Among them, ∈ represents the noise at time step t, x t is the input state, is the adjacent frame constraint, the adapter's parameters are updated in the following way:

[0046] Loss (θ k )

[0047] Where Θ={φ adapter ,φ unshared ,T share} denotes the parameters of the adapter, non-shared tokens, and shared token embeddings, η is the learning rate, The loss is the gradient of the loss function with respect to Θ.

[0048] Preferably, the step S3 specifically includes:

[0049] S31: Backward diffusion step. The probability framework in the back diffusion process uses the following formula to calculate the noise input x t Perform iterative denoising:

[0050]

[0051] Among them, μ θ (x t ,t) is the predicted mean, Σ θ (t) represents the uncertainty adjustment variance in the backward step, and the discrete-time backward diffusion step is expressed as:

[0052]

[0053] Among them, x t is the noise input at time step t, ∈ θ (x t ,t) is the noise predicted by the model, α t is the time-dependent variance, is the cumulative product of the variance from the start to time step t, and z is the random noise sampled from the standard normal distribution;

[0054] S32: Bilateral filtering step, the potential representation x t Apply bilateral filtering, the output of bilateral filtering on pixel x is O x Defined as:

[0055]

[0056] in, represents the set of neighborhood pixels around x, y represents each pixel in the area, and its influence on x is based on the proximity of space and intensity. x and I y where x and y are the intensity values ​​of pixels, respectively. The spatial and intensity components are defined as follows:

[0057]

[0058] Among them, σ spatial Controls the filter’s sensitivity to spatial distance, σ intensity The filter smoothes the image by taking both spatial proximity and intensity similarity into account, thereby preserving edge details while reducing noise.

[0059] On the other hand, the present invention also provides a lightweight text-generated video device, which receives input text and applies a diffusion model of a 2D UNet architecture. The device includes the following modules:

[0060] U-Net timing adapter module, which is used to integrate lightweight adapter modules in U-Net for timing fine-tuning;

[0061] The semantic alignment adaptation module uses a combination of shared token embedding of global context and dynamic non-shared token embedding for frame-specific information to achieve semantic alignment;

[0062] The DDIM video inversion module is used to integrate bilateral filtering that considers both spatial proximity and intensity similarity into the DDIM framework to achieve temporally coherent video output.

[0063] Preferably, the U-Net timing adaptation module specifically includes:

[0064] The U-Net image reconstruction module uses the U-Net model to predict noise in each denoising step to gradually reconstruct a clear image.

[0065] x t+1 =x t +ε t -θ(x t ,t)

[0066] Among them, x t+1 represents the image at step t, ε t To predict noise, θ is the U-Net model;

[0067] The intermediate feature map extraction module registers hooks for each time step to extract intermediate feature maps:

[0068]

[0069] Among them, x is the input feature, B l,b With A l,b is a learnable lightweight low-rank matrix parameter;

[0070] Temporal consistency loss representation module, temporal loss function Reduce the inconsistency of feature representation by minimizing the change in similarity between frames:

[0071]

[0072] The similarity function of adjacent feature maps is:

[0073]

[0074] where F t and F t+1 represents the feature maps extracted at consecutive time steps t and t+1 respectively;

[0075] Diffusion loss characterization module, the standard diffusion loss is defined as follows:

[0076]

[0077] Objective function generation module, the objective function combines the temporal consistency loss and diffusion loss:

[0078]

[0079] where λ temporal and λ diffusion They are set to 1 and 0.01 respectively to balance the contribution of each loss term.

[0080] Preferably, the semantic alignment adaptation module specifically includes:

[0081] Visual hidden feature extraction module: for the input image I, the visual encoder of the CLIP model extracts the visual hidden feature H vision ∈R B×N×d , where B is the batch size, N = 50 is the sequence length, and d = 768 is the feature dimension;

[0082] The non-shared subset generation module takes the average of every two adjacent feature vectors on the second dimension to generate the non-shared subset H sub ∈R B×N / 2×d , whose expression is:

[0083]

[0084] for

[0085] Apply the projection matrix W∈R d×d To H sub , to maintain the feature dimension:

[0086] Z unshare =H sub W,W = nn.Linear(d,d);

[0087] Text embedding module,Text embedding is used for time-series-aware fine-tuning and is constructed as:

[0088]

[0089] Among them, T share is a shared token embed, From the normal distribution Initialization, Z frame is a frame-specific non-shared token, is the concatenation operation of conditional and unconditional embedding in the first dimension of the sequence, and [·] represents concatenation in the sequence dimension.

[0090] Preferably, the alignment adaptation module further includes:

[0091] Training update module,During the training process, the parameters (θ) of the CLIP visual encoder are kept frozen, and only the adapter parameters, the projection layer (φ) of shared embeddings and non-shared tokens are iteratively optimized to minimize the following loss function:

[0092]

[0093] Among them, ∈ represents the noise at time step t, x t is the input state, is the adjacent frame constraint, the adapter's parameters are updated in the following way:

[0094] Loss (θ k )

[0095] Where Θ={φ adapter ,φ unshared ,T share} denotes the parameters of the adapter, non-shared tokens, and shared token embeddings, η is the learning rate, The loss is the gradient of the loss function with respect to Θ.

[0096] Preferably, the DDIM video inversion module specifically includes:

[0097] Backward diffusion module, the probability framework in the back diffusion process uses the following formula to calculate the noise input x t Perform iterative denoising:

[0098]

[0099] Among them, μ θ (x t ,t) is the predicted mean, Σ θ (t) represents the uncertainty adjustment variance in the backward step, and the discrete-time backward diffusion step is expressed as:

[0100]

[0101] Among them, x t is the noise input at time step t, ∈ θ (x t ,t) is the noise predicted by the model, α t is the time-dependent variance, is the cumulative product of the variance from the start to time step t, and z is the random noise sampled from the standard normal distribution;

[0102] Bilateral filtering module, for the potential representation x t Apply bilateral filtering, the output of bilateral filtering on pixel x is O x Defined as:

[0103]

[0104] in, represents the set of neighborhood pixels around x, y represents each pixel in the area, and its influence on x is based on the proximity of space and intensity. x and I y where x and y are the intensity values ​​of pixels, respectively. The spatial and intensity components are defined as follows:

[0105]

[0106] Among them, σ spatial Controls the filter’s sensitivity to spatial distance, σ intensity The filter smoothes the image by taking both spatial proximity and intensity similarity into account, thereby preserving edge details while reducing noise.

[0107] The present invention provides a lightweight text-to-video method and device, which combines prompt word learning and time-aware DDIM inversion technology to improve the text-to-image video editing performance based on the diffusion algorithm with the lowest training cost. The method improves the temporal consistency, semantic alignment, inter-frame consistency and video quality in the text-to-video generation framework without additional training.

[0108] Compared with the prior art, the present invention provides a lightweight text-to-video method and device, which has the following beneficial effects:

[0109] 1. This method has temporal and semantic constraints and does not require training any existing spatial or temporal modules;

[0110] 2. Bilateral filtering is introduced in video inversion technology. This method effectively alleviates the inconsistency between frames by smoothing the noise latent variables, preserving the structural edges and ensuring the temporal coherence.

[0111] 3. Extensive experiments show that our lightweight adapter improves the efficiency by more than 50% compared with mainstream T2V models (such as AnimatedDiff’s 2200M training weights) with only 860M trainable parameters on average, and significantly enhances the temporal consistency, semantic alignment, and video quality of the T2I-based T2V algorithm;

[0112] 4. This method is universal and compatible, and can seamlessly support T2I-based T2V algorithms and ControlNet modules, such as edge detection, depth maps, and human posture estimation, which can meet both general purposes and customized applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0113] Figure 1 A schematic diagram of a lightweight text-to-video method disclosed in the present invention;

[0114] Figure 2 A schematic diagram of the lightweight text-to-video model framework process disclosed in the present invention;

[0115] Figure 3 It is a flow chart of the semantic alignment adaptation framework disclosed in the present invention;

[0116] Figure 4 Comparison of video generation quality for different algorithms with and without the adapter of the present invention;

[0117] Figure 5 Comparison of user evaluations of the video generation algorithms with and without the adapter of the present invention. DETAILED DESCRIPTION

[0118] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0119] Attached Figure 1 A lightweight method for generating videos from text is shown in the figure below. The method inputs text and applies it to a diffusion model of a 2D U-Net architecture. The method includes the following steps:

[0120] S1: U-Net timing adaptation step, integrating a lightweight adapter module in U-Net for timing fine-tuning;

[0121] S2: semantic alignment adaptation step, which combines shared token embedding of global context with dynamic non-shared token embedding for frame-specific information to achieve semantic alignment;

[0122] S3: DDIM video inversion step, which integrates bilateral filtering that considers both spatial proximity and intensity similarity into the DDIM framework to achieve temporally coherent video output.

[0123] Figure 2 The schematic diagram of the lightweight text generation video model framework flow chart is shown. The present invention trains a model based on Stable Diffusion v1.5, combining a U-Net with 860M parameters and a text encoder with 123M parameters. A ControlNet designed for deep data is integrated, and a jointly trained PrefixToken module is equipped to enhance video-prompt alignment. Mixed precision (fp16) is used during training, the learning rate is 3e-5, and the input frame resolution is 512x512. The 1.3G parameter ControlNet requires a dedicated training phase, and four RTX4090 GPUs are used for continuous training for 20 hours. The PrefixToken module itself has 2.3M parameters. The module is trained together with the U-Net adapter, and the joint training takes 18 hours, using only one RTX4090 GPU. The training of ControlNet and Shared Token uses the same hyperparameters as above.

[0124] Diffusion models in video generation face significant challenges in maintaining temporal consistency. Although the U-Net architecture in StableDiffusion performs well in generating high-quality single-frame images, it lacks the ability to handle temporal dependencies between consecutive frames, often resulting in inconsistencies between frames, such as flickering, which affects the coherence and authenticity of the generated video. To address these problems, the present invention proposes a U-Net temporal adapter that connects adjacent frames by introducing a constraint mechanism to enhance temporal consistency. This method is based on the basic process of U-Net, which predicts noise in each denoising step to gradually reconstruct a clear image:

[0125] x t+1 =x t +∈ t -θ(x t ,t)

[0126] Among them, x t+1 represents the image of step t, ∈ t To predict noise, θ is a U-Net model. Although this mechanism is very effective in generating static images, it does not inherently capture the temporal relationship between consecutive frames. To address this limitation, the Temporal U-Net Adapter integrates a lightweight adapter module in U-Net for efficient temporal fine-tuning. First, a hook is registered for each time step to extract the intermediate feature map:

[0127]

[0128] Among them, x is the input feature, B l,b With A l,b is a learnable lightweight low-rank matrix parameter.

[0129] In order to achieve smooth transition between frames, a similarity function of adjacent feature maps is defined:

[0130]

[0131] where F t and F t+1 Represents the feature maps extracted at consecutive time steps t and t+1 respectively. Based on this, the temporal loss function is defined Reduce the inconsistency of feature representation by minimizing the change in similarity between frames:

[0132]

[0133] In addition, to ensure smooth and natural transitions during the denoising process, a standard diffusion loss is also required:

[0134]

[0135] The final objective function combines temporal consistency loss and diffusion loss:

[0136]

[0137] where λ temporal and λ diffusion They are set to 1 and 0.01 respectively to balance the contribution of each loss term.

[0138] Existing video editing algorithms have difficulty aligning frames and text semantics due to the static nature of the embedding method, the segmented integration of conditional and unconditional text, and the neglect of the dynamic role of text. Figure 3 A schematic flowchart of the semantic alignment adaptation framework is presented, which introduces a unified framework that combines shared token embeddings for global context with dynamic non-shared token embeddings for frame-specific information.

[0139] Shared Token Embed From the normal distribution For the input image I, the visual encoder of the CLIP model extracts the visual hidden features H vision ∈R B×N×d , where B is the batch size, N = 50 is the sequence length, and d = 768 is the feature dimension. The unshared subset H is generated by averaging every two adjacent feature vectors in the second dimension. sub ∈R B×N / 2×d , whose expression is:

[0140]

[0141] for

[0142] Next, apply the projection matrix W∈R d×d To H sub , to maintain the feature dimension:

[0143] Z unshare =H sub W,W=nn.Linear(d,d)

[0144] The final text embedding is used for timing-aware fine-tuning and is constructed as:

[0145]

[0146] Among them, T share is the shared token embedding, Z frame is a frame-specific non-shared token, is the concatenation operation of the conditional and unconditional embeddings in the first dimension of the sequence, and [·] represents the concatenation in the sequence dimension. During training, the parameters (θ) of the CLIP visual encoder are kept frozen, and only the adapter parameters, the projection layer (φ) of the shared embeddings and the unshared tokens are iteratively optimized to minimize the following loss function:

[0147]

[0148] Among them, ∈ represents the noise at time step t, x t is the input state, is the adjacent frame constraint. The adapter's parameters are updated as follows:

[0149] Loss (θ k )

[0150] Where Θ={φ adapter ,φ unshared ,T share} denotes the parameters of the adapter, non-shared tokens, and shared token embeddings, η is the learning rate, The loss is the gradient of the loss function with respect to Θ. Note that the adapter (φ adapter ) is active only during the training time range 0.5 to 1.0; otherwise, it remains inactive.

[0151] Existing DDIM-based video inversion techniques face challenges in ensuring frame-level smoothness and quality. The randomness of the diffusion process often introduces uneven textures, noise artifacts, and detail loss, resulting in visually inconsistent frames, including texture breaks and loss of clarity. These issues stem from the probabilistic framework of the back-diffusion process, which is based on the following formula for the noisy input x t Perform iterative denoising:

[0152]

[0153] Among them, μ θ (x t ,t) is the predicted mean, Σ θ (t) represents the uncertainty adjustment variance in the backward step. The discrete-time backward diffusion step is expressed as:

[0154]

[0155] Among them, x t is the noise input at time step t, ∈ θ (x t ,t) is the noise predicted by the model, α t is the time-dependent variance, is the cumulative product of the variance from the start to time step t, and z is the random noise sampled from the standard normal distribution. In our improved DDIM video inversion algorithm, we have the latent representation x t A bilateral filter is applied to improve temporal consistency and reduce noise artifacts between frames. This filter smoothes the image by taking into account both spatial proximity and intensity similarity, thereby preserving edge details while reducing noise. The output O of the bilateral filter for pixel x is x Defined as:

[0156]

[0157] in, represents the set of neighborhood pixels around x, and y represents each pixel in the region, whose influence on x is based on spatial and intensity proximity. x and I y are the intensity values ​​of pixel x and y, respectively. The spatial and intensity components are defined as follows:

[0158]

[0159] Among them, σ spatial Controls the filter’s sensitivity to spatial distance, σ intensity The filter is conditioned to respond to intensity differences. Integrating bilateral filtering into the DDIM video inversion framework ensures that each time step modifies the latent representation x t Perform smoothing to generate x t ′ This approach maintains structural integrity and visual consistency across frames, resulting in temporally coherent video output.

[0160] In addition, the present invention also provides a lightweight text-generated video device, which receives input text and applies a diffusion model of a 2D U-Net architecture. The device includes the following modules:

[0161] U-Net timing adapter module, which is used to integrate lightweight adapter modules in U-Net for timing fine-tuning;

[0162] The semantic alignment adaptation module uses a combination of shared token embedding of global context and dynamic non-shared token embedding for frame-specific information to achieve semantic alignment;

[0163] The DDIM video inversion module is used to integrate bilateral filtering that considers both spatial proximity and intensity similarity into the DDIM framework to achieve temporally coherent video output.

[0164] Preferably, the U-Net timing adaptation module specifically includes:

[0165] The U-Net image reconstruction module uses the U-Net model to predict noise in each denoising step to gradually reconstruct a clear image.

[0166] x t+1 =x t +∈ t -θ(x t ,t)

[0167] Among them, x t+1 represents the image of step t, ∈ t To predict noise, θ is the U-Net model;

[0168] The intermediate feature map extraction module registers hooks for each time step to extract intermediate feature maps:

[0169]

[0170] Among them, x is the input feature, B l,b With A l,b is a learnable lightweight low-rank matrix parameter;

[0171] Temporal consistency loss representation module, temporal loss function Reduce the inconsistency of feature representation by minimizing the change in similarity between frames:

[0172]

[0173] The similarity function of adjacent feature maps is:

[0174]

[0175] where F t and F t+1 represents the feature maps extracted at consecutive time steps t and t+1 respectively;

[0176] Diffusion loss characterization module, the standard diffusion loss is defined as follows:

[0177]

[0178] Objective function generation module, the objective function combines the temporal consistency loss and diffusion loss:

[0179]

[0180] where λ temporal and λ diffusion They are set to 1 and 0.01 respectively to balance the contribution of each loss term.

[0181] Preferably, the semantic alignment adaptation module specifically includes:

[0182] Visual hidden feature extraction module: for the input image I, the visual encoder of the CLIP model extracts the visual hidden feature H vision ∈R B×N×d , where B is the batch size, N = 50 is the sequence length, and d = 768 is the feature dimension;

[0183] The non-shared subset generation module takes the average of every two adjacent feature vectors on the second dimension to generate the non-shared subset H sub ∈R B×N / 2×d , whose expression is:

[0184]

[0185] for Apply the projection matrix W∈R d×d To H sub , to maintain the feature dimension:

[0186] Z unshare =H sub W,W = nn.Linear(d,d);

[0187] Text embedding module,Text embedding is used for time-series-aware fine-tuning and is constructed as:

[0188]

[0189] Among them, T share is a shared token embed, From the normal distribution Initialization, Z frame is a frame-specific non-shared token, is the concatenation operation of conditional and unconditional embedding in the first dimension of the sequence, and [·] represents concatenation in the sequence dimension.

[0190] Preferably, the alignment adaptation module further includes:

[0191] Training update module,During the training process, the parameters (θ) of the CLIP visual encoder are kept frozen, and only the adapter parameters, the projection layer (φ) of shared embeddings and non-shared tokens are iteratively optimized to minimize the following loss function:

[0192]

[0193] Among them, ∈ represents the noise at time step t, x t is the input state, is the adjacent frame constraint, the adapter's parameters are updated in the following way:

[0194] Loss (θ k )

[0195] Where Θ={φ adapter ,φ unshared ,T share} denotes the parameters of the adapter, non-shared tokens, and shared token embeddings, η is the learning rate, The loss is the gradient of the loss function with respect to Θ.

[0196] Preferably, the DDIM video inversion module specifically includes:

[0197] Backward diffusion module, the probability framework in the back diffusion process uses the following formula to calculate the noise input x t Perform iterative denoising:

[0198]

[0199] Among them, μ θ (x t ,t) is the predicted mean, Σ θ (t) represents the uncertainty adjustment variance in the backward step, and the discrete-time backward diffusion step is expressed as:

[0200]

[0201] Among them, x t is the noise input at time step t, ∈ θ (x t ,t) is the noise predicted by the model, α t is the time-dependent variance, is the cumulative product of the variance from the start to time step t, and z is the random noise sampled from the standard normal distribution;

[0202] Bilateral filtering module, for the potential representation x t Apply bilateral filtering, the output of bilateral filtering on pixel x is O x Defined as:

[0203]

[0204] in, represents the set of neighborhood pixels around x, y represents each pixel in the area, and its influence on x is based on the proximity of space and intensity. x and I y where x and y are the intensity values ​​of pixels, respectively. The spatial and intensity components are defined as follows:

[0205]

[0206] Among them, σ spatial Controls the filter’s sensitivity to spatial distance, σ intensityThe filter smoothes the image by taking both spatial proximity and intensity similarity into account, thereby preserving edge details while reducing noise.

[0207] The present invention uses the MSR-VTT dataset, which contains 10,000 video clips across 20 categories, each with 20 English subtitles, a total of about 29,000 unique words, and the dataset is divided into 6,513 training clips, 497 validation clips, and 2,990 test clips. By using OpenAI's ChatGPT API, the subtitles are modified to generate new content, and a specialized DataLoader is used to batch adjacent frames for early denoising, converting MP4 videos to WebDataset format to optimize training. During inference, the adapter weights are loaded into the StableDiffusionPipeline and integrated with the model at a ratio of 50%.

[0208] The present invention conducts a comparative study with the existing methods on the MSR-VTT dataset through quantitative, qualitative comparison and manual evaluation. The comparison of the video generation algorithm before and after the injection of the adapter shows a significant improvement in perceptual quality and text-image alignment. From the quantitative results, the adapter significantly improves the performance of all models: Text2Video-Zero's LPIPS drops from 0.402 to 0.319, and CLIP increases from 28.82 to 31.38, ensuring smoother transitions and better prompt following; TokenFlow's FID drops significantly from 166.72 to 143.21, while LPIPS and CLIP scores increase, enhancing motion and texture consistency; Vid2Vid's LPIPS drops from 0.303 to 0.253, and CLIP increases from 31.94 to 32.78, reflecting the improvement in motion alignment; VidToMe's LPIPS drops from 0.126 to 0.114, and CLIP increases from 25.00 to 28.12, indicating improvements in perceptual quality and semantic alignment. From the visual effect, Figure 4 We show that these improvements are equally significant. Without the adapter, the videos often suffer from motion inconsistencies, artifacts, and weak temporal consistency, such as in scenes like a lion roaring or a child riding through water. In contrast, with the adapter, temporal stability, motion consistency, and semantic alignment with cues are improved, especially in complex scenes like surfing or dog walking, where transitions are smoother and more realistic. The adapter’s contribution lies in its ability to enhance fidelity and consistency between frames.

[0209] User Evaluation We invited 67 participants to evaluate the video generation model trained on the MSR-VTT dataset. Each participant watched video clips of the four algorithms with and without our fine-tuned adapter, without being informed of the existence of the adapter, and evaluated the text-image alignment, image quality, and consistency of the videos. Figure 5 A comparison of user evaluations of video generation algorithms with and without adapters is shown. Adapter-enhanced videos are more popular among all algorithms, especially in terms of consistency in VidToMe and image quality in TokenFlow, which significantly improves the overall alignment and frame quality.

[0210] The present invention discloses a lightweight text-to-video method and apparatus that utilizes a pre-trained diffusion model to enhance temporal consistency and visual quality with minimal additional training. The adapter is a versatile, plug-and-play solution that reduces flicker and improves text-to-image alignment, and can be seamlessly integrated into existing workflows.

[0211] Those skilled in the art may appreciate that the units of the various examples described in conjunction with the embodiments disclosed in the present invention may be implemented in the form of electronic hardware, or a combination of computer software and electronic hardware.

[0212] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

Claims

1. A lightweight text-to-video method, which takes text as input and applies it to a diffusion model of a 2D U-Net architecture, characterized in that: The method comprises the following steps: S1: U-Net timing adaptation step, integrating a lightweight adapter module in U-Net for timing fine-tuning; S2: semantic alignment adaptation step, which combines shared token embedding of global context with dynamic non-shared token embedding for frame-specific information to achieve semantic alignment; S3: DDIM video inversion step, which integrates bilateral filtering that considers both spatial proximity and intensity similarity into the DDIM framework to achieve temporally coherent video output.

2. The method for generating video from text according to claim 1, characterized in that: The step S1 specifically includes: S11: U-Net image reconstruction step, using the U-Net model to predict noise in each denoising step to gradually reconstruct a clear image. x t+1 =x t +∈ t -θ(x t ,t) Among them, x t+1 represents the image of step t, ∈ t To predict noise, θ is the U-Net model; S12: Intermediate feature map extraction step, a hook is registered for each time step to extract the intermediate feature map: Among them, x is the input feature, B l,b With A l,b is a learnable lightweight low-rank matrix parameter; S13: Temporal consistency loss characterization step, timing loss function Reduce the inconsistency of feature representation by minimizing the change in similarity between frames: The similarity function of adjacent feature maps is: where F t and F t+1 represents the feature maps extracted at consecutive time steps t and t+1 respectively; S14: Diffusion loss characterization step, the standard diffusion loss is defined as follows: S15: Objective function generation step, the objective function combines the temporal consistency loss and the diffusion loss: where λ temporal and λ diffusion They are set to 1 and 0.01 respectively to balance the contribution of each loss term.

3. The method for generating video from text according to claim 1, characterized in that: The step S2 specifically includes: S21: Visual hidden feature extraction step. For the input image I, the visual encoder of the CLIP model extracts the visual hidden feature H vision ∈R B×N×d , where B is the batch size, N = 50 is the sequence length, and d = 768 is the feature dimension; S22: Non-shared subset generation step, averaging every two adjacent feature vectors on the second dimension to generate a non-shared subset H sub ∈R B×N / 2×d , whose expression is: for Apply the projection matrix W∈R d×d To H sub , to maintain the feature dimension: Z unshare =H sub ·W,W=nn.Linear(d,d); S23: Text embedding step. Text embedding is used for time-series-aware fine-tuning and is constructed as: Among them, T share is a shared token embed, From the normal distribution Initialization, Z frame is a frame-specific non-shared token, is the concatenation operation of conditional and unconditional embedding in the first dimension of the sequence, and [·] represents concatenation in the sequence dimension.

4. The method for generating video from text according to claim 3, characterized in that: The step S2 further comprises: S24: Training update step, during the training process, the parameters (θ) of the CLIP visual encoder remain frozen, and only the adapter parameters, the projection layer (φ) of the shared embedding and the non-shared tokens are iteratively optimized to minimize the following loss function: Among them, ∈ represents the noise at time step t, x t is the input state, is the adjacent frame constraint, the adapter's parameters are updated in the following way: Where Θ={φ adapter ,φ unshared ,T share } denotes the parameters of the adapter, non-shared tokens, and shared token embeddings, η is the learning rate, The loss is the gradient of the loss function with respect to Θ.

5. The method for generating video from text according to claim 1, characterized in that: The step S3 specifically includes: S31: Backward diffusion step. The probability framework in the back diffusion process uses the following formula to calculate the noise input x t Perform iterative denoising: Among them, μ θ (x t ,t) is the predicted mean, Σ θ (t) represents the uncertainty adjustment variance in the backward step, and the discrete-time backward diffusion step is expressed as: Among them, x t is the noise input at time step t, ∈ θ (x t ,t) is the noise predicted by the model, α t is the time-dependent variance, is the cumulative product of the variance from the start to time step t, and z is the random noise sampled from the standard normal distribution; S32: Bilateral filtering step, the potential representation x t Apply bilateral filtering, the output of bilateral filtering on pixel x is O x Defined as: in, represents the set of neighborhood pixels around x, y represents each pixel in the area, and its influence on x is based on the proximity of space and intensity. x and I y where x and y are the intensity values ​​of pixels, respectively. The spatial and intensity components are defined as follows: Among them, σ spatial Controls the filter’s sensitivity to spatial distance, σ intensity Adjusts the filter's response to intensity differences. The filter smoothes the image by taking both spatial proximity and intensity similarity into account, thereby preserving edge details while reducing noise.

6. A lightweight text-to-video device, which receives input text and applies a diffusion model based on a 2D UNet architecture, characterized in that: The device includes the following modules: U-Net timing adapter module, which is used to integrate lightweight adapter modules in U-Net for timing fine-tuning; The semantic alignment adaptation module uses a combination of shared token embedding of global context and dynamic non-shared token embedding for frame-specific information to achieve semantic alignment; The DDIM video inversion module is used to integrate bilateral filtering that considers both spatial proximity and intensity similarity into the DDIM framework to achieve temporally coherent video output.

7. The text-to-video device according to claim 6, characterized in that: The U-Net timing adaptation module specifically includes: The U-Net image reconstruction module uses the U-Net model to predict noise in each denoising step to gradually reconstruct a clear image. x t+1 =x t +∈ t -θ(x t ,t) Among them, x t+1 represents the image of step t, ∈ t To predict noise, θ is the U-Net model; The intermediate feature map extraction module registers hooks for each time step to extract intermediate feature maps: Among them, x is the input feature, B l,b With A l,b is a learnable lightweight low-rank matrix parameter; Temporal consistency loss representation module, temporal loss function Reduce the inconsistency of feature representation by minimizing the change in similarity between frames: The similarity function of adjacent feature maps is: where F t and F t+1 represents the feature maps extracted at consecutive time steps t and t+1 respectively; Diffusion loss characterization module, the standard diffusion loss is defined as follows: Objective function generation module, the objective function combines the temporal consistency loss and diffusion loss: where λ temporal and λ diffusion They are set to 1 and 0.01 respectively to balance the contribution of each loss term.

8. The text-to-video device according to claim 6, characterized in that: The semantic alignment adaptation module specifically includes: Visual hidden feature extraction module: for the input image I, the visual encoder of the CLIP model extracts the visual hidden feature H vision ∈R B×N×d , where B is the batch size, N = 50 is the sequence length, and d = 768 is the feature dimension; The non-shared subset generation module takes the average of every two adjacent feature vectors on the second dimension to generate the non-shared subset H sub ∈R B×N / 2×d , whose expression is: for Apply the projection matrix W∈R d×d To H sub , to maintain the feature dimension: Z unshare =H sub ·W,W=nn.Linear(d,d); Text embedding module,Text embedding is used for time-series-aware fine-tuning and is constructed as: Among them, T share is a shared token embed, From the normal distribution Initialization, Z frame is a frame-specific non-shared token, is the concatenation operation of conditional and unconditional embedding in the first dimension of the sequence, and [·] represents concatenation in the sequence dimension.

9. The text-to-video device according to claim 8, characterized in that: The alignment adaptation module further includes: Training update module,During the training process, the parameters (θ) of the CLIP visual encoder are kept frozen, and only the adapter parameters, the projection layer (φ) of shared embeddings and non-shared tokens are iteratively optimized to minimize the following loss function: Among them, ∈ represents the noise at time step t, x t is the input state, is the adjacent frame constraint, the adapter's parameters are updated in the following way: Where Θ={φ adapter ,φ unshared ,T share } denotes the parameters of the adapter, non-shared tokens, and shared token embeddings, η is the learning rate, The loss is the gradient of the loss function with respect to Θ.

10. The text-to-video device according to claim 6, characterized in that: The DDIM video inversion module specifically includes: Backward diffusion module, the probability framework in the back diffusion process uses the following formula to calculate the noise input x t Perform iterative denoising: Among them, μ θ (x t ,t) is the predicted mean, Σ θ (t) represents the uncertainty adjustment variance in the backward step, and the discrete-time backward diffusion step is expressed as: Among them, x t is the noise input at time step t, ∈ θ (x t ,t) is the noise predicted by the model, α t is the time-dependent variance, is the cumulative product of the variance from the start to time step t, and z is the random noise sampled from the standard normal distribution; Bilateral filtering module, for the potential representation x t Apply bilateral filtering, the output of bilateral filtering on pixel x is O x Defined as: in, represents the set of neighborhood pixels around x, y represents each pixel in the area, and its influence on x is based on the proximity of space and intensity. x and I y where x and y are the intensity values ​​of pixels, respectively. The spatial and intensity components are defined as follows: Among them, σ spatial Controls the filter’s sensitivity to spatial distance, σ intensity Adjusts the filter's response to intensity differences. The filter smoothes the image by taking both spatial proximity and intensity similarity into account, thereby preserving edge details while reducing noise.

Images