Text-driven video generation method based on single video fine tuning

Abstract

The invention relates to the technical field of video generation and machine learning, and provides a text-driven video generation method based on single video fine tuning, which comprises the following steps: encoding an input video into potential feature representation through a pre-training encoder, and constructing input features; generating a video frame by adopting a video generation network based on a diffusion model, and optimizing the generation process of the video frame by adopting a pre-trained CLIP model and a pre-trained VGG model; and based on the dynamic loss weight adjustment strategy, adaptively balancing the weight of the semantic and texture constraint, and optimizing the training process of the CLIP model and the VGG model. According to the method provided by the invention, a multi-scale space adapter, semantic features based on CLIP and texture features based on VGG are combined. Through a progressive training strategy and dynamic loss weight balance, the quality and consistency of generated frames are improved, text alignment, time coherence and generalization ability are remarkably improved, and the development of the text to video generation field is promoted.

Description

Technical Field

[0001] This invention relates to the fields of video generation and machine learning technology, and more specifically, to a text-driven video generation method based on single-video fine-tuning. Background Technology

[0002] With the rapid development of AIGC, diffusion models have become a significant breakthrough in the field of generative models. Although traditional generative models such as generative adversarial networks have achieved remarkable results in image generation tasks, they often face problems such as instability and mode collapse during training.

[0003] Diffusion models have made significant progress in image synthesis tasks; however, video generation still faces many challenges, primarily the dual requirements of high-quality frames and temporal coherence. Previous research has introduced semantic consistency into image diffusion generation, using CLIP guidance to improve resolution performance. The VDM model models using 3D convolution and spatiotemporal attention mechanisms, but still suffers from instability during training. Zhang et al. pointed out that decoupling semantics and texture helps improve frame quality; however, its joint optimization for temporal coherence remains unresolved.

[0004] Traditional methods utilize Generative Adversarial Networks (GANs) to achieve realistic video generation through adversarial training between the generator and discriminator. However, due to the temporal dimension of video data, GANs often face difficulties in maintaining spatial and temporal consistency. To address this issue, methods such as MoCoGAN and TGAN introduce temporal modeling mechanisms, but problems such as mode collapse and insufficient temporal consistency still exist. Existing research extends the diffusion process to the temporal dimension, gradually restoring high-quality and temporally consistent videos through noise. Although current text-to-video (T2V) models have achieved significant results, their performance largely depends on training on large-scale video datasets. Furthermore, simultaneously optimizing semantic and texture feature losses is a complex challenge in video generation tasks. Semantic features focus on capturing the consistency of content and context, while texture features focus on visual details. Dynamic weighting strategies achieve balanced optimization by adjusting the weights of semantic and texture losses during training. Mao et al. proposed a time-step-based linear weighting strategy, initially prioritizing semantic consistency and then gradually shifting towards texture consistency. However, existing dynamic weighting methods often neglect the detail frequency information that is crucial for balancing semantic and texture losses. Summary of the Invention

[0005] In view of this, the present invention proposes a text-driven video generation method based on single-video fine-tuning. It can effectively adapt to the pre-trained text-to-image (T2I) diffusion model with only a text-video pair, and achieve efficient T2V generation. It calculates the semantic distance between the generated frame and the target frame through feature alignment, improves cross-frame texture consistency through temporal consistency constraints, optimizes texture consistency by constraining the similarity between video frames, and uses spectral information to adaptively adjust the loss weights to improve the stability of optimization and the quality of the generated video, thereby solving the problems existing in the above-mentioned prior art.

[0006] To achieve the above objectives, this invention proposes a text-driven video generation method based on single-video fine-tuning, characterized by comprising: The input video is encoded into a latent feature representation by a pre-trained encoder to construct the input features; A video generation network based on a diffusion model is used to generate video frames, and the generation process of the video frames is optimized using pre-trained CLIP and VGG models. The training process of the CLIP model and VGG model is optimized by adaptively balancing the weights of semantic and texture constraints based on a dynamic loss weight adjustment strategy.

[0007] Furthermore, during the process of generating video frames using a video generation network based on a diffusion model, the input features are processed using a multi-scale spatial adapter to fuse features of different scales into the input features, thereby constructing fused features.

[0008] Furthermore, the multi-scale spatial adapter structure is represented by the following equation:

[0009] in, For the input feature matrix, and There are two sets of downsampling layers. and There are two sets of upsampling layers.

[0010] Furthermore, the multi-scale spatial adapter adopts a lightweight design, only inserting a small number of parameters into the pre-trained model for fine-tuning.

[0011] Furthermore, the process of generating video frames using a diffusion-based video generation network includes: The input features enter the forward diffusion process, where noise is gradually added to simulate feature degradation, and the original data is recovered through the reverse diffusion process. The ResNet module is expanded into a 3D convolutional block to perform the reverse diffusion process. In each reverse diffusion process, the denoised video frame is input into the pre-trained CLIP and VGG models respectively to extract semantic and texture features.

[0012] Furthermore, in the process of generating video frames using a diffusion-based video generation network, an attention mechanism module is introduced. This module models intra-frame dependencies through spatial context attention, aligns external semantic information through cross-modal attention, and captures inter-frame temporal consistency through temporal attention.

[0013] Furthermore, the optimization process for the generation of the video frames using dual supervision signals from different pre-trained networks includes: The CLIP model is used to extract semantic features from images and text. The semantic distance between the generated frame and the target frame is optimized by mean square error. The VGG model is used to extract multi-level texture features, and the texture features between the video frame and the target frame are compared.

[0014] Furthermore, the weighting process for adaptively balancing semantic and texture constraints based on a dynamic loss weight adjustment strategy includes: Fourier transform is applied to downsampled image representations at different scales to decompose the image into amplitude spectrum and phase spectrum; The weights of semantic loss and texture loss are dynamically adjusted in the loss function based on the relative spectral energy of semantic and texture components. The semantic loss and texture loss are calculated by comparing the predicted output with the Fourier transform results of the corresponding real frame, and the semantic loss and texture loss are combined into a consistency loss.

[0015] Furthermore, the consistency loss is expressed as follows: = +

[0016]

[0017] in, Indicates consistency loss. Indicates semantic loss, Indicates texture loss, Indicates dynamic weights. and These represent the spectral energy of semantic loss and texture loss, respectively.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention proposes a novel framework—Dual Consistency Training (DCT)—aimed at jointly optimizing semantic and texture consistency in the video generation process. Specifically, it introduces a multi-scale spatial adapter to enhance spatial feature extraction capabilities and combines the complementary advantages of CLIP and VGG: CLIP focuses on high-level semantic representation, while VGG captures fine-grained texture and image details. During the training phase, a staged strategy is employed to apply semantic and texture losses separately to constrain the differences between generated and real frames. Furthermore, this invention proposes a CLWS mechanism, which dynamically adjusts the weights between semantic and texture losses to achieve a more stable and efficient optimization process. Notably, DCT requires only a single training video and an NVIDIA A6000 graphics card to achieve high-quality video generation. Extensive experimental results demonstrate that the proposed method significantly improves temporal coherence and visual fidelity in multiple video generation tasks, validating its effectiveness and versatility. Attached Figure Description

[0019] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. In the drawings: Figure 1 This is a schematic diagram illustrating the overall structure and principle of the DCT framework proposed in this invention; Figure 2 is a schematic diagram of the video generation method proposed in this invention, wherein (a) is text-guided video editing, and (b) is text-to-video generation; Figure 3 This is a schematic diagram of the multi-scale space adapter structure in an embodiment of the present invention; Figure 4 is a visual comparison diagram of the method of the present invention with five baseline models, where (a) is the CogVideo model, (b) is the Plug-and-Play model, (c) is the Text2LIVE model, (d) is the AnimateDiff model, (e) is the T2V-Zero model, and (f) is the DCT model highlighted in the present invention. Figure 5 This is a qualitative comparison chart of CLWS under four weighting strategies in the embodiments of the present invention; Figure 6 The diagram shows the ablation experiment results in the embodiments of the present invention, where (a) is the result of the complete DCT model, (b) is the result lacking semantic consistency, (c) is the result lacking texture consistency, and (d) is the result lacking a multi-scale spatial adapter. Detailed Implementation

[0020] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0021] This embodiment proposes a text-driven video generation method based on single-video fine-tuning, including the following steps: The input video is encoded into a latent feature representation by a pre-trained encoder to construct the input features; A video generation network based on a diffusion model is used to generate video frames, and the generation process of the video frames is optimized using pre-trained CLIP and VGG models. The training process of the CLIP model and VGG model is optimized by adaptively balancing the weights of semantic and texture constraints based on a dynamic loss weight adjustment strategy.

[0022] Based on the above steps, the technical solution and principle of this embodiment will be described in detail: Diffusion models are an important branch of generative model research in recent years. Their core idea is to utilize data... The data is converted into random noise, and the original data is gradually recovered through a reverse process. In this section, this embodiment will introduce the forward diffusion process, the reverse restoration process, and its training objective of the diffusion model.

[0023] The forward diffusion process gradually transmits data through a Markov chain. It is converted into Gaussian noise. Specifically, the data is processed... Noise is gradually added at each time step, and the transition probability at each step is: in =1,..., It is a time step; This is a hyperparameter that controls the noise intensity at each step, and it is usually increased incrementally during training, such as linearly or cosinely. After... After a step, the data will gradually evolve into standard Gaussian noise. At any time step ,data It can be represented as raw data. Linear weighted combination of noise: in It is a time step The cumulative scaling factor, This represents noise sampled from a standard Gaussian distribution.

[0024] The goal of the reverse diffusion process is to gradually reconstruct each step of the noise introduction process during the forward diffusion process, from the noise... Generate raw data The conditional probability of the reverse process is:

[0025] in: It is the mean value estimated by the neural network; It is the covariance matrix (usually fixed as a diagonal matrix).

[0026] The core training objective of the diffusion model is to use neural networks Noise introduced during forward diffusion By simplifying the formula, the training objective can be defined as the mean squared error (MSE):

[0027] The Dual Consistency Training (DCT) framework proposed in this embodiment aims to address a key challenge in text-to-video generation: maintaining semantic alignment between the generated video and the input text while ensuring temporal continuity of visual textures across frames. To comprehensively address these challenges, DCT constructs an end-to-end pipeline comprising three core components. First, a multi-scale spatial adapter processes the input features at multiple resolutions to capture a combination of coarse and fine spatial information. Next, the framework utilizes dual supervision signals from different pre-trained networks: a CLIP-based semantic supervision branch ensures semantic consistency between the generated content and the text, while a VGG-based texture supervision branch enhances the preservation of detailed visual information. These two complementary losses jointly guide the diffusion-based video generation network, producing semantically faithful and temporally coherent video frames. Finally, the framework introduces a dynamic loss weight adjustment strategy (CLWS) to adaptively balance the weights of semantic and texture constraints during training. This overall design enables DCT to learn efficiently with minimal data, achieving high-quality video generation with strong temporal consistency.

[0028] The model proposed in this embodiment is built based on a pre-trained stable diffusion framework, and its overall architecture is as follows: Figure 1 As shown. During training, each action is represented by a video clip and its corresponding text prompt. Video frames are first encoded into latent feature representations by a pre-trained encoder. X iThe model then proceeds to the forward diffusion process, gradually adding noise to simulate feature degradation. The reverse diffusion process uses an extended U-Net network to remove noise and restore clean features. To capture the temporal dependencies between video frames, this embodiment extends the ResNet module to a 3D convolutional block and introduces a multi-scale spatial adapter to enhance feature representation. Currently, the model primarily targets single-object scenes; future plans include introducing temporal attention mechanisms and instance segmentation to address the challenges of scene complexity and temporal dependencies arising from multi-object interactions. In each reverse diffusion step, the denoised frames are input into pre-trained CLIP and VGG models respectively to extract semantic and texture features. During training, the denoising loss, semantic consistency loss, and texture consistency loss are jointly optimized.

[0029] exist Figure 1 Given a text-video pair (e.g., "A jeep is driving on the highway"), this method generates text-to-video (T2V) based on a pre-trained text-to-image (T2I) diffusion model. The framework integrates three attention modules: spatial context attention to model intra-frame dependencies, cross-modal attention to align external semantic information (such as CLIP features), and temporal attention to capture inter-frame temporal consistency. Some experimental results are shown in Figure 2.

[0030] Multi-scale space adapter: Large-scale text-to-image (T2I) pre-trained models demonstrate superior performance in personalized generation and image editing tasks, exhibiting significant transferability. Therefore, lightweight fine-tuning methods can effectively utilize spatial information in video generation. Inspired by efficient fine-tuning techniques in natural language processing and vision tasks, this embodiment selects the adapter method due to its simplicity and efficiency.

[0031] In the DCT framework of this embodiment, the core objective of the multi-scale spatial adapter is to enhance the representational power of input features by fusing features from different scales. Specifically, as... Figure 3 As shown, the adapter first uses two sets of fully connected layers to downsample and upsample the input features, extracting features at different scales, denoted as... ^, and then reconstruct it back to the original input dimension. This method not only enhances the multi-scale information of features, but also effectively captures global and local spatial information. The multi-scale spatial adapter can be represented as:

[0032] Downsampling and upsampling processes: Input feature matrix Passing through two sets of downsampling layers in sequence and and two sets of upsampling layers and This allows for feature extraction and reconstruction at different scales. Downsampling helps the model encode input information at multiple scales, while upsampling helps recover and enhance higher-level feature representations.

[0033] The interaction of multi-scale features: In diffusion models, the input data undergoes multiple rounds of denoising. The role of the multi-scale spatial adapter is to fuse features of different scales into the original input features in each generation round, thereby enriching the diffusion process. Specifically, this fusion of multi-scale features helps the diffusion model maintain the global structure while capturing local details when processing fine-grained spatial information.

[0034] This module employs a lightweight design, inserting only a small number of parameters into the pre-trained model for fine-tuning, avoiding large-scale adjustments to the main model and significantly reducing parameter updates and computational overhead. Simultaneously, the multi-scale spatial adapter efficiently integrates features from different scales, enhancing the expressive power of spatial information and promoting faster training convergence. Its simple structure and computational efficiency further reduce resource consumption and time costs. Overall, this design effectively improves training efficiency and shortens training time while maintaining performance.

[0035] Semantic texture consistency Semantic-texture consistency is a key objective in video generation, aiming to ensure that the generated content is semantically and visually consistent with the target. The CLIP model is employed to extract semantic features from images and text, and the semantic distance between generated and target frames is optimized using mean squared error (MSE). Texture consistency is particularly important for visual quality and detail preservation, especially in dynamic scenes. Therefore, a VGG network is used to extract multi-level texture features. By comparing the texture features between generated and target frames, visual consistency is ensured and detail representation is improved.

[0036] Consistency Loss Weighted Scheduling (CLWS): In computer vision, images contain not only spatial information but also rich frequency domain features. Traditional weighting schemes, such as Gaussian or exponential functions, use fixed weights and lack adaptability to frequency features, making it difficult to effectively capture structural and detailed information in complex scenes. Fourier transform decomposes images into low-frequency and high-frequency components, where low frequencies represent global structure and high frequencies encode texture and detail information. Utilizing the Fourier spectrum allows for dynamic adjustment of the weights of semantic and texture losses, achieving an adaptive balance between structural and detail features during training. This content-aware mechanism outperforms fixed-weight methods, significantly improving the model's stability and performance in complex visual tasks.

[0037] 1) Multiscale Fourier Transform To capture multi-level frequency features, this embodiment applies different scales. The downsampled image representation applies a Fourier transform. For the scaled image... Its frequency is calculated as follows:

[0038] in, It is an image In scale A downsampled version. By calculating the frequency domain transform at multiple scales, information at different frequency levels can be captured. Amplitude and phase fusion. 2) Amplitude and phase fusion Amplitude and phase information are crucial for image reconstruction. The loss function simultaneously integrates the amplitude and phase spectra, allowing the optimization process to consider both energy distribution (amplitude) and spatial structure (phase). The Fourier transform decomposes the signal into its amplitude spectrum. Phase spectrum Its decomposition form can be expressed as:

[0039] 3) Adaptive weight adjustment Based on frequency domain characteristics, especially the energy distribution of the spectrum, we use frequency domain energy distribution... and The weights of semantic loss and texture loss are dynamically adjusted in the loss function.

[0040]

[0041] in, and These represent the Fourier transform results of the semantic image and the texture image, respectively.

[0042] Based on these energy distributions, adaptive weights Defined as:

[0043] 4) Consistency Loss Function To ensure that the generated video frames are consistent with the target in terms of semantic content and fine-grained details, we define two frequency domain loss terms: semantic loss. and texture loss These losses are calculated by comparing the predicted output with the Fourier transform of the corresponding ground truth frame. Semantic and texture losses together measure the frequency domain consistency between the generated output and the target frame. Specifically, the loss function is defined as follows:

[0044] in, This represents the semantic or texture features extracted from the generated frame in the frequency domain. This corresponds to the target spectrum. Semantic loss emphasizes the consistency of global content, while texture loss focuses on preserving local structural details, such as edges and fine textures. These two objectives are complementary and are subsequently merged into a unified consistency loss. To effectively integrate these two objectives, this embodiment represents the unified consistency loss as a weighted sum: = +

[0045] Weight ∈[0,1] is adaptively calculated based on the relative spectral energy of semantic and texture components. This dynamic weighting mechanism enables the model to automatically prioritize consistency across different types during training, based on the features of the current generated results. As training progresses, the model gradually achieves a balance between structural alignment and detail synthesis, thereby improving the semantic fidelity and visual realism of the generated frames.

[0046] Comparative experiment: This experiment is based on a latent variable diffusion model, trained using a single motion video and its corresponding text description. We uniformly sample 24 frames from the video at a resolution of 512×512 and use a learning rate of 3×10⁻⁶. -5 The batch size was 1, and 500 steps of fine-tuning were performed using a random seed of 33. Mixed-precision training (FP16) was enabled during training to reduce memory usage and accelerate training. The DDIM sampler was used during inference, without classifier guidance. Fine-tuning a single video took approximately 34 minutes. All experiments were performed on a single A6000 GPU, with approximately 15GB of VRAM used during training and approximately 8GB during inference. Brief scalability tests showed that training time increased approximately linearly with frame rate and approximately quadratically with resolution; for example, for 48 frames per second and a resolution of 768×768, the training time was approximately 75 minutes. Mixed-precision training effectively ensured practical training efficiency.

[0047] Dataset: To evaluate the proposed method, 25 videos were selected from the DAVIS dataset. DAVIS is a widely used benchmark dataset in video analysis, renowned for its high-quality, diverse, and well-annotated content. Although originally designed for object segmentation tasks, its rich motion patterns and complex scenes make it well-suited for text-driven video generation. Videos were selected from three representative categories: animals, vehicles, and people. For each video, four cue words were designed, covering object editing, background modification, and style transformation, resulting in a total of 100 evaluation samples.

[0048] During DCT model training, a video was randomly selected from the corresponding video set as a template for each motion mode. This random selection ensured diversity during training and improved the model's generalization ability. Subsequently, this experiment compared the model with five publicly available baseline models and provided a detailed performance evaluation. 1) CogVideo: A T2V model trained on a dataset of 5.4 million videos with captions. It can generate high-quality videos directly from text prompts through zero-shot learning, demonstrating powerful text-to-video generation capabilities.

[0049] 2) Plug-and-Play: This is an advanced image editing mode that allows users to edit each frame of a video individually, greatly enhancing the flexibility of video content editing.

[0050] 3) Text2LIVE: This method uses hierarchical neural maps for text-guided video editing, which can efficiently convert complex text descriptions into high-quality dynamic video editing.

[0051] 4) AnimateDiff: Focuses on generating continuous and high-quality dynamic frame animations through text descriptions, which can transform static images into smooth animation sequences, thereby improving the conversion effect of images to videos.

[0052] 5) Text2Video-Zero: This model does not rely on a large amount of training data or labeled dynamic samples. It generates videos directly through natural language descriptions, demonstrating its powerful capabilities in zero-shot learning scenarios.

[0053] Table 1

[0054] Quantitative results: We evaluated our method against five baseline models based on text alignment, inter-frame continuity, and generation diversity, as shown in Table 1 (Quantitative comparison of our method with text-to-video generation methods).

[0055] This experiment evaluates the matching degree between video and text description by calculating the CLIP score for each frame and averaging it. The CLIP model is used to measure the semantic relevance between images and text; a higher score indicates stronger semantic consistency between the generated video and the input text. Next, we evaluate frame consistency by calculating the cosine similarity between the CLIP image embeddings of all frame pairs. Inter-frame continuity reflects the smoothness of the transition between frames during video generation; a higher cosine similarity indicates smoother frame connections in the video. Finally, we use the CLIP model to extract image embeddings of all frames to represent the entire video and calculate the cosine distance between videos to evaluate the diversity of generated videos. A smaller cosine distance indicates lower similarity between videos, i.e., higher generation diversity. Experimental results show that CogVideo can generate consistent video frames but struggles to express text descriptions; while Plug-and-Play performs well in inter-frame continuity, but the generated video content is limited in terms of variation; Text2LIVE can better reflect the semantics of the input text; AnimateDiff needs improvement in fluency; and T2V-Zero performs well in text alignment. In comparison, the method of this invention outperforms the baseline in both metrics.

[0056] Table 2

[0057] The CLWS strategy was evaluated using peak signal-to-noise ratio (PSNR) and optical flow consistency metrics, and the results are shown in Table 2. The Mul-Fourier algorithm outperformed other methods due to its fast convergence speed and high-quality output. The Gaussian method performed poorly in terms of motion consistency and image quality, while the Fourier and Exponential methods were more stable, but their performance was slightly inferior to that of the Mul-Fourier method.

[0058] The qualitative results are illustrated in Figure 4, which depicts a cartoon-style scene of Spider-Man skiing on the beach, visually comparing the proposed method with five baseline models. Observations show that CogVideo outputs poor video quality and fails to reflect the general concept of the text; Plug-and-Play lacks continuity between video frames; Text2LIVE struggles to accurately represent the content of the edited text; the AnimateDiff method cannot be combined with the high-performance but heterogeneous T2I model, and while it performs well in terms of consistency and diversity, it has limitations in texture alignment; although T2V-Zero generates visually appealing frames, it performs poorly in generating videos with reasonable motion patterns. In contrast, our method demonstrates superior performance in terms of inter-frame continuity, text alignment, and video quality.

[0059] like Figure 5As shown, the four weighting strategies exhibit different effects in the vehicle region. Gaussian weighting over-smooths the image, resulting in blurred edges and loss of high-frequency details; Exponential weighting improves smoothness but still causes deformation of the vehicle structure; the single Fourier method enhances some frequency components but lacks edge sharpness. In contrast, the Mul-Fourier strategy effectively fuses multi-scale frequency information, generating sharper edges, richer textures, and better structural consistency.

[0060] This embodiment further conducted an ablation experiment to evaluate the importance of each module. The text prompt was "A puppy is eating an orange," as shown below. Figure 6 As shown in (b), there is an improvement in complexity and detail, but the subject is not prominent enough and the picture is too complicated, which affects the overall visual appeal and the clarity of semantic expression. Figure 6 (c) Due to increased complexity and unnatural handling of details, the overall effect is not clear and concise enough, lacking the prominence of the theme and the balance of the composition. Figure 6 (d) The theme is not prominent enough, and the naturalness and compositional balance are poor. These results demonstrate the significant contribution of each key module to the final complete model. As shown in Table 3, the complete model shows significant improvements in all three metrics, where MS represents the multi-scale spatial adapter, S represents semantic consistency, and T represents texture consistency.

[0061] Table 3

[0062] Table 4 compares different visual model methods, showing that although ViT and BLIP improve consistency scores, they significantly increase memory usage and training time. In contrast, DCT, while maintaining similar performance, significantly reduces memory usage and improves training efficiency. Therefore, considering both effectiveness and efficiency, DCT demonstrates a more practical advantage.

[0063] Table 4

[0064] In summary, this embodiment proposes a text-driven video generation method based on single-video fine-tuning—Dual Consistency Training (DCT). This method combines a multi-scale spatial adapter with CLIP-based semantic features and VGG-based texture features. Through a progressive training strategy and dynamic loss weight balancing (CLWS), the quality and consistency of generated frames are improved. Experimental results verify the significant improvements in text alignment, temporal coherence, and generalization ability, thus advancing the field of text-to-video generation.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A text-driven video generation method based on single-video fine-tuning, characterized in that, include: The input video is encoded into a latent feature representation by a pre-trained encoder to construct the input features; A video generation network based on a diffusion model is used to generate video frames, and the generation process of the video frames is optimized using pre-trained CLIP and VGG models. The training process of the CLIP model and VGG model is optimized by adaptively balancing the weights of semantic and texture constraints based on a dynamic loss weight adjustment strategy.

2. The text-driven video generation method based on single-video fine-tuning according to claim 1, characterized in that, In the process of generating video frames using a diffusion-based video generation network, the input features are processed at multiple resolutions using a multi-scale spatial adapter, and features at different scales are fused into the input features to construct fused features.

3. The text-driven video generation method based on single-video fine-tuning according to claim 1, characterized in that, The multi-scale space adapter structure is represented by the following formula: ， in, For the input feature matrix, and There are two sets of downsampling layers. and There are two sets of upsampling layers.

4. The text-driven video generation method based on single-video fine-tuning according to claim 1, characterized in that, The multi-scale spatial adapter adopts a lightweight design, only inserting a small number of parameters into the pre-trained model for fine-tuning.

5. The text-driven video generation method based on single-video fine-tuning according to claim 1, characterized in that, The process of generating video frames using a diffusion-based video generation network includes: The input features enter the forward diffusion process, where noise is gradually added to simulate feature degradation, and the original data is recovered through the reverse diffusion process. The ResNet module is expanded into a 3D convolutional block to perform the reverse diffusion process. In each reverse diffusion process, the denoised video frame is input into the pre-trained CLIP and VGG models respectively to extract semantic and texture features.

6. The text-driven video generation method based on single-video fine-tuning according to claim 1, characterized in that, In the process of generating video frames using a diffusion-based video generation network, an attention mechanism module is introduced. This module models intra-frame dependencies through spatial context attention, aligns external semantic information through cross-modal attention, and captures inter-frame temporal consistency through temporal attention.

7. The text-driven video generation method based on single-video fine-tuning according to claim 1, characterized in that, The optimization process for the generation of the video frames using dual supervision signals from different pre-trained networks includes: The CLIP model is used to extract semantic features from images and text. The semantic distance between the generated frame and the target frame is optimized by mean square error. The VGG model is used to extract multi-level texture features, and the texture features between the video frame and the target frame are compared.

8. The text-driven video generation method based on single-video fine-tuning according to claim 1, characterized in that, The process of adaptively balancing semantic and texture constraints based on a dynamic loss weight adjustment strategy includes: Fourier transform is applied to downsampled image representations at different scales to decompose the image into amplitude spectrum and phase spectrum; The weights of semantic loss and texture loss are dynamically adjusted in the loss function based on the relative spectral energy of semantic and texture components. The semantic loss and texture loss are calculated by comparing the predicted output with the Fourier transform results of the corresponding real frame, and the semantic loss and texture loss are combined into a consistency loss.

9. The text-driven video generation method based on single-video fine-tuning according to claim 8, characterized in that, The consistency loss is shown in the following formula: = + ， ， in, Indicates consistency loss. Indicates semantic loss, Indicates texture loss, Indicates dynamic weights. and These represent the spectral energy of semantic loss and texture loss, respectively.

Images