Text generation video diffusion model training method and device, equipment and medium

Abstract

The application relates to the technical field of artificial intelligence and multimedia generation, in particular to a text-to-video diffusion model training method and device, equipment and a medium. The method comprises the following steps: obtaining a training data set of a text-to-video diffusion model; generating a first video sample according to a real video in the training data set, obtaining a second video sample after the first video sample is subjected to noise increase and removal, generating a video sample pair according to the first video sample and the second video sample; correcting a video time artifact of a target time period in the video sample pair, determining a loss function of the text-to-video diffusion model according to a preference optimization target of the video diffusion model, and training the text-to-video diffusion model based on the corrected video sample and the loss function. Thus, the problems of incoherent AI-generated video timing and local picture distortion in related technologies are solved.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and multimedia generation technology, and in particular to a training method, apparatus, device and medium for a text-to-video diffusion model. Background Technology

[0002] Text-to-video diffusion models often employ preference alignment for post-training optimization. These techniques primarily rely on manual annotation or external reward models to obtain preference signals, which is prone to introducing bias and is costly. Furthermore, video temporal defects are often concentrated in short local segments. Conventional methods apply equal supervision to all video frames, leading to an imbalance in resource allocation and making it difficult to effectively correct visual imperfections such as motion flickering and temporal discontinuities within the video sequence. Summary of the Invention

[0003] This application provides a training method, apparatus, device, and medium for a text-generated video diffusion model to solve problems such as discontinuous timing and distortion of local images in AI (Artificial Intelligence) generated videos in related technologies.

[0004] The first aspect of this application provides a training method for a text-generated video diffusion model, comprising the following steps: obtaining a training dataset for the text-generated video diffusion model; generating a first video sample based on real videos in the training dataset; obtaining a second video sample by adding and removing noise from the first video sample; generating video sample pairs based on the first video sample and the second video sample; correcting video temporal artifacts in the video sample pairs for the target time period; determining the loss function of the text-generated video diffusion model based on the preference optimization objective of the video diffusion model; and training the text-generated video diffusion model based on the corrected video samples and the loss function.

[0005] Based on the above technical means, this application embodiment uses real video to add noise and denoise to reconstruct video sample pairs, and generates supervision signals by relying on the model's own reconstruction bias, without relying on manual annotation and external reward models, thus reducing training costs and external biases; at the same time, it locates target time periods with temporal defects and specifically corrects temporal artifacts, concentrating optimization resources on segments prone to defects, and avoiding normal frames diluting the training gradient; it combines the preference optimization objective to construct a loss function for model training, effectively improving the problems of flickering and temporal breaks in the generated video, and enhancing the temporal coherence and visual realism of the text-generated video.

[0006] Optionally, generating a first video sample based on real videos in the training dataset includes: inputting real videos from the training dataset into a pre-trained video encoder; generating a first latent variable through the video encoder; and using the first latent variable as the first video sample.

[0007] Based on the above technical means, the embodiments of this application use a pre-trained video encoder to map real video to the latent space to obtain the first latent variable and use it as the first video sample. Subsequent noise addition and reconstruction operations are carried out in the low-dimensional latent variable space, reducing the amount of data computation of the original video image; at the same time, the video representation form is unified, which facilitates the subsequent calculation of error differences between samples and improves the stability of the sample processing flow.

[0008] Optionally, the second video sample is obtained by adding and removing noise from the first video sample, including: applying forward diffusion to the first video sample to obtain a second latent variable; performing back diffusion for the remaining steps starting from the second latent variable to obtain a third latent variable; and using the third latent variable as the second video sample.

[0009] According to the above technical means, the embodiments of this application add noise to the first latent variable corresponding to the first video sample through forward diffusion, and then perform the denoising and reconstruction process through back diffusion to obtain the third latent variable as the second video sample. Reconstructed samples are generated by relying on the denoising dynamics of the diffusion model itself. The difference between the real video and the reconstructed sample can reflect the trajectory error in the model inference process, and provide a data foundation that fits the real inference behavior of the model for the subsequent construction of video sample pairs.

[0010] Optionally, generating a video sample pair based on the first video sample and the second video sample includes: extracting the first frame of the first video sample; replacing the first frame of the second video sample with the first frame of the first video sample; generating a positive sample based on the first video sample, generating a negative sample based on the second video sample after replacing the first frame of the first video sample, and generating a video sample pair based on the positive sample and the negative sample.

[0011] Based on the above technical means, in this application embodiment, the first frame of the first video sample is used to replace the first frame of the second video sample to construct a video sample pair composed of positive and negative samples. This eliminates the irrelevant deviation between positive and negative samples caused by the difference in the first frame, avoids the model focusing its training attention on the difference in the first frame, and prompts the model to focus on learning the temporal defects in the subsequent frames of the video sequence. This allows the constructed video sample pair to more accurately reflect the temporal artifact differences in the middle segments of the video.

[0012] Optionally, before correcting the video time artifacts in the target time period in the video sample pair, the method further includes: extracting the first latent variable of the positive samples and the third latent variable of the negative samples in the video sample pair; calculating the squared error of the video sample pair frame by frame based on the first latent variable and the third latent variable; calculating the average difference of each consecutive time window based on the squared error; and taking the time window with the largest average difference as the target time period.

[0013] Based on the above technical means, the embodiments of this application calculate the squared error of positive and negative samples frame by frame in the latent space, and use a sliding window to statistically analyze the average difference of each continuous time period. The continuous time window with the highest error is automatically selected as the target time period, which accurately locates the local segments in the video that are prone to temporal artifacts. This provides a clear range for subsequent centralized optimization, prevents optimization resources from being occupied by normal images, and improves the pertinence of temporal defect correction.

[0014] Optionally, the loss function of the text-to-video diffusion model is determined based on the preference optimization objective of the video diffusion model, including: determining the video sample loss of the video diffusion model based on the preference optimization objective; calculating the model loss of the video diffusion model relative to the reference model based on the video sample loss; calculating the preference loss of the video diffusion model based on the model loss; and generating the loss function of the text-to-video diffusion model based on the preference loss.

[0015] Based on the above technical means, the embodiments of this application sequentially calculate the video sample loss, the model loss of the current model relative to the reference model, and the preference loss, and construct a loss function based on the preference loss. The reference model is used as a stable benchmark to measure the model reconstruction deviation, quantify the effect difference between positive and negative samples, form a preference supervision target that is adapted to the time series centralized optimization scenario, and guide the model to iteratively update in the direction of reducing the difference in time series defects.

[0016] Optionally, the expression for the loss function of the preference optimization objective is:

[0017] in, To optimize the loss for centralized implicit preferences, Based on real video and corresponding text conditions Seeking expectations, It is the sigmoid (S-type activation function) function. The preference is logit (log odds value); the expression for the preference logit is:

[0018] in, , To preset hyperparameters, It is a rectifier linear unit. This represents the difference in diffusion reconstruction error between the video diffusion model and the reference model. To extract positive sample segments after the target time period, This refers to the negative sample segment obtained after extracting the target time period.

[0019] Based on the above technical means, the embodiments of this application adopt centralized implicit preference optimization loss as the training objective, rely on preference logit to quantify the reconstruction error difference between positive and negative sample segments within the target time period, and at the same time use ReLU (Rectified Linear Unit) term to set stable regularization constraints to suppress excessive model updates that destroy the original effective generation capability; through expectation operation combined with real video and text conditions to carry out model optimization, guide the model to specifically reduce the temporal feature differences within the target time period and improve the local video temporal distortion problem.

[0020] A second aspect of this application provides a training apparatus for a text-generated video diffusion model, comprising: an acquisition module for acquiring a training dataset for the text-generated video diffusion model; a generation module for generating a first video sample based on real videos in the training dataset, obtaining a second video sample by adding and removing noise from the first video sample, and generating video sample pairs based on the first video sample and the second video sample; and a correction module for correcting video temporal artifacts in the video sample pairs for a target time period, determining a loss function for the text-generated video diffusion model based on the preferred optimization objective of the video diffusion model, and training the text-generated video diffusion model based on the corrected video samples and the loss function.

[0021] Optionally, the generation module is further configured to: input real videos from the training dataset into a pre-trained video encoder; generate a first latent variable through the video encoder; and use the first latent variable as a first video sample.

[0022] Optionally, the generation module is further configured to: apply forward diffusion to the first video sample to obtain a second latent variable; perform back diffusion for the remaining steps starting from the second latent variable to obtain a third latent variable; and use the third latent variable as the second video sample.

[0023] Optionally, the generation module is further configured to: extract the first frame of the first video sample; replace the first frame of the second video sample with the first frame of the first video sample; generate a positive sample based on the first video sample, generate a negative sample based on the second video sample after the first frame of the first video sample has been replaced, and generate a video sample pair based on the positive sample and the negative sample.

[0024] Optionally, it also includes: a calculation module, used to extract the first latent variable of the positive sample and the third latent variable of the negative sample in the video sample pair before correcting the video time artifacts of the target time period in the video sample pair; calculate the squared error of the video sample pair frame by frame based on the first latent variable and the third latent variable; calculate the average difference of each consecutive time window based on the squared error; and take the time window with the largest average difference as the target time period.

[0025] Optionally, the correction module is further configured to: determine the video sample loss of the video diffusion model based on the preference optimization objective; calculate the model loss of the video diffusion model relative to the reference model based on the video sample loss; calculate the preference loss of the video diffusion model based on the model loss; and generate the loss function of the video diffusion model based on the preference loss.

[0026] Optionally, the expression for the loss function of the preference optimization objective is:

[0027] in, To optimize the loss for centralized implicit preferences, Based on real video and corresponding text conditions Seeking expectations, For the sigmoid function, Prefers logit; The expression for preferring logit is:

[0028] in, , To preset hyperparameters, It is a rectifier linear unit. This represents the difference in diffusion reconstruction error between the video diffusion model and the reference model. To extract positive sample segments after the target time period, This refers to the negative sample segment obtained after extracting the target time period.

[0029] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the program to implement the training method for a text-generated video diffusion model as described in the above embodiments.

[0030] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement a training method for a text-generated video diffusion model as described in the above embodiments.

[0031] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0032] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1This is a flowchart illustrating a training method for a text-generated video diffusion model according to an embodiment of this application; Figure 2 This is a system architecture diagram provided according to an embodiment of this application; Figure 3 This is a block diagram of a training apparatus for a text-generated video diffusion model according to an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0033] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0034] Currently, text-to-video generation technology based on diffusion models has made significant progress and is widely used in multimedia intelligent generation scenarios. Mainstream technologies often employ preference alignment and post-training optimization to improve video generation quality. However, these technologies generally suffer from two major drawbacks, as detailed below: First, there is a bottleneck in preference attribution. Existing preference signals mainly rely on manual annotation or external reward models for acquisition. Manual annotation is costly and data updates are lagging, making it difficult to adapt to the dynamic learning process of the model. External reward models are prone to introducing inherent biases, leading to optimization instability and reward hacking problems, and it is difficult to capture the cumulative error of the model's inference denoising trajectory.

[0035] Second, there is a problem of incorrect allocation of time credit. Video temporal artifacts are mostly local, short-term, and occasional defects, but existing optimization methods apply supervision evenly to all video frames, diluting the effective gradient with normal frames and resulting in insufficient optimization for local faulty segments.

[0036] In summary, the preference optimization schemes of related text-to-video diffusion models generally rely on external preference signals, have unreasonable resource allocation in the time dimension, and are difficult to accurately correct temporal defects in videos while balancing model training stability and the perceptual realism of video generation.

[0037] The following describes the training method, apparatus, device, and medium of the text-generated video diffusion model according to embodiments of this application, with reference to the accompanying drawings. Addressing the problems mentioned in the background art, such as discontinuous temporal sequences and distortions in local scenes in AI-generated videos, this application provides a training method for a text-generated video diffusion model, namely the cIPO (concentrated implicit preference optimization) scheme. This method innovates from two dimensions: implicit preference construction and time-concentrated optimization. It abandons the preference acquisition method relying on manual annotation or external reward models. Instead, it leverages the model's own denoising dynamics, utilizing the reconstruction bias generated by forward denoising of real videos and backward denoising reconstruction to construct positive and negative preference sample pairs, obtaining implicit preference signals that accurately reflect the model's inference trajectory errors. Simultaneously, a time-concentrated optimization strategy is designed to automatically locate the continuous time window with the largest video reconstruction error as the defect target period, concentrating the preference optimization loss on local fault segments and avoiding the gradient dilution problem caused by global uniform supervision.

[0038] This solves the problems of high cost of acquiring preference signals and easy deviation in related technologies for video preference optimization, as well as unreasonable resource allocation in time dimension optimization and poor effect of local short-term artifact correction. It effectively improves the defects of motion collapse, screen flickering and time sequence breakage in generated video, and significantly improves the temporal coherence and visual perception realism of text-generated video while retaining the original effective content of the video.

[0039] Specifically, Figure 1 This is a flowchart illustrating a training method for a text-generated video diffusion model provided in an embodiment of this application.

[0040] like Figure 1 As shown, the training method for this text-to-video diffusion model includes the following steps: In step S101, the training dataset for the text-to-video diffusion model is obtained.

[0041] Among them, the text-to-video diffusion model is a generation model built based on a diffusion mechanism to achieve the task of generating videos from text.

[0042] Text-to-video diffusion models typically consist of four main modules: (1) a text encoder that maps input prompts to conditional embeddings c; (2) a spatiotemporal VAE that compresses the video into a latent space z0=E(x)∈RC×T×H×W via encoder E, and reconstructs it by decoder after denoising; (3) a diffusion backbone (early 3DU-Net, currently DiT is the mainstream), which performs inverse denoising Φθ in the latent space and fuses text conditions through cross-attention; and (4) a diffusion scheduler that controls the forward denoising and inverse iteration trajectories. The experiments in this paper are based on Wan-VACE, which uses DiT as the backbone and Wan-VAE as the spatiotemporal compressor, and processes conditional inputs through a unified Video Conditioning Unit in T2V mode. It is understood that the embodiments of this application collect training datasets for the text-to-video generation task, providing real and effective raw data support for subsequent sample construction, error calculation and model training, and ensuring the reliability and accuracy of subsequent implicit preference signal construction and temporal defect localization.

[0043] Specifically, the training dataset collected in this application embodiment contains several real videos. This application can subsequently construct implicit preference pairs based on real videos in the training dataset, and then carry out model preference alignment training.

[0044] In step S102, a first video sample is generated based on the real videos in the training dataset. The first video sample is then processed by adding and removing noise to obtain a second video sample. A video sample pair is generated based on the first video sample and the second video sample.

[0045] It is understood that the embodiments of this application construct positive and negative video sample pairs based on the model's own dynamic process of adding noise and reconstructing real videos. Without the need for manual annotation and external reward models, implicit preference data that fits the model's inference of the real trajectory error can be obtained. At the same time, through the sample pair construction method, basic sample conditions are provided for subsequent temporal error comparison and local defect optimization.

[0046] It should be noted that the real videos are the original video footage from the training dataset; the first video sample is the optimized sample obtained by encoding the real video. The second video sample is the initial non-optimal sample obtained by reconstructing the first video sample after adding noise and reverse denoising. Video sample pairs are preference sample pairs consisting of positive and negative samples.

[0047] Specifically, this step involves implicit preference pair construction based on Rollout reconstruction (trajectory extrapolation). It utilizes the model's own denoising dynamics to construct preference pairs, replacing manual annotation or external reward models. The complete process is as follows: (1) Latent space coding of real videos: For real videos in the training set Latent variables are obtained through a pre-trained video VAE (Variational Autoencoder) encoder E. ; (2) Forward noise addition: Select the initial diffusion step ,right By applying a forward diffusion process, some noisy latent variables are obtained. ; (3) Model inverse denoising and reconstruction: from To begin, use the currently trainable policy model. In text conditions Perform backdiffusion rollout for the remaining steps under constraints to obtain the terminal reconstruction latent variables. ; (4) Define implicit preference pairs: the original clean latent variables As a preferred sample This will reconstruct latent variables As a non-preferred sample ; (5) First frame anchoring: To reduce first frame drift and structural collapse, use First frame replacement The first frame yields the anchored negative sample. This method directly constructs preference pairs from the model's denoising trajectory error. The preference signals have the characteristics of rollout-aware (reflecting inference error) and self-grounded (derived from the model's own dynamics), eliminating the need for manual or external reward models and reducing costs and bias.

[0048] In addition to fixing the initial diffusion step, the embodiments of this application are as follows: Alternatively, random sampling s~Uniform(1, N-1) can be used, or adaptive selection can be made based on video and text prompts. Besides anchoring the first frame, anchoring can be done on several previous frames or structured keyframes, or anchoring can be omitted; in addition to complete reconstruction, negative samples can be truncated midway through rollout to obtain multi-stage negative samples, or multiple negative samples can be constructed based on different noise levels to form a list-style preference; all of the above schemes can construct implicit preference pairs of "real video as preferred sample and video recovered from noise by the model as non-preferred sample" to achieve the same effect.

[0049] The diffusion model completes video generation through two basic processes: forward denoising and backward denoising. The original real video samples are within the video data distribution range, and after progressive denoising, they are mapped to a high-noise space. Theoretically, the model can completely restore the original video content by removing noise along a standard path. However, due to the model's performance limitations, the actual denoising path deviates from the ideal trajectory. The deviation between the two paths continues to accumulate, resulting in temporal defects such as timing misalignment, screen flickering, and body distortion in the final output video.

[0050] This application relies on mining implicit reward signals from real video data and continuously optimizes the denoising process of the correction model through implicit reward training. This reduces the deviation between the actual denoising path and the ideal denoising path, eliminates accumulated temporal errors, and thus improves the temporal coherence problem in the video generation process, thereby enhancing the final video generation quality. Figure 2 As shown in step one of the modules, the complete process of encoding the real video, forward noise addition, and progressive reverse denoising by the model is performed to obtain reconstructed latent variables, constructing preference sample pairs composed of preferred and non-preferred samples. Traditional DPO baseline methods rely on offline video generation and manual annotation, which suffers from preference attribution bottlenecks, distribution mismatches, and temporal credit misallocation defects. The cIPO scheme in this application generates implicit rewards based on real data, eliminating the need for manual annotation and guiding video generation online, thus alleviating the distribution bias problem.

[0051] In this embodiment of the application, generating a first video sample based on real videos in the training dataset includes: inputting real videos from the training dataset into a pre-trained video encoder; generating a first latent variable through the video encoder; and using the first latent variable as the first video sample.

[0052] It is understood that the embodiments of this application use a pre-trained video encoder to map real video to the latent space to obtain the first latent variable and use it as the first video sample. Subsequent noise addition and reconstruction operations are carried out in the low-dimensional latent variable space, which reduces the amount of data computation of the original video image. At the same time, the video representation form is unified, which facilitates the subsequent calculation of error differences between samples and improves the stability of the sample processing flow.

[0053] It should be noted that the pre-trained video encoder is a pre-trained video VAE encoder, which is an encoding module that has been trained in advance and is used to map the original video to the latent space; the first latent variable is the latent space representation of the real video after being encoded by the video encoder. .

[0054] Specifically, the training set will include real videos. Input a pre-trained video VAE encoder E to encode latent variables. The first latent variable As the first video sample.

[0055] In this embodiment of the application, the process of obtaining a second video sample by adding and removing noise from a first video sample includes: applying forward diffusion to the first video sample to obtain a second latent variable; performing back diffusion for the remaining number of steps starting from the second latent variable to obtain a third latent variable; and using the third latent variable as the second video sample.

[0056] It is understood that in this embodiment, noise is added to the first latent variable corresponding to the first video sample through forward diffusion, and then the third latent variable is obtained as the second video sample by performing the denoising and reconstruction process through back diffusion. The reconstruction sample is generated by relying on the denoising dynamics of the diffusion model itself. The difference between the real video and the reconstruction sample can reflect the trajectory error in the model inference process, and provide a data foundation that fits the real inference behavior of the model for the subsequent construction of video sample pairs.

[0057] It should be noted that forward diffusion is a diffusion process in which noise is gradually added to the latent variable; the second latent variable is the part of the latent variable with added noise obtained after forward diffusion of the first latent variable. Backdiffusion is the rollout process of a diffusion model that progressively removes noise to achieve video reconstruction; the third latent variable is the terminal reconstruction latent variable output after completing all remaining backdiffusion steps. .

[0058] Specifically, the initial diffusion step is selected. For the first latent variable (Originated from real video via VAE encoder E, i.e.) Apply forward diffusion: sample noise from a standard Gaussian distribution The noise is then added to the predefined noise-adding function of the scheduler S. By time step By mixing the preset coefficients, a portion of the noisy latent variables are obtained. That is, the second latent variable; based on the current trainable policy model (The video diffusion backbone, corresponding to the T2V generation path of Wan-VACE in the paper) acts as the denoising executor, under the constraint of text condition c, from Start executing the remaining steps (i.e., from step s to step N out of the total N steps in the scheduler). A 1-step backdiffusion rollout, with a total of N steps. Backdiffusion at each time step (s-step): The model is based on The input is prediction noise (or velocity field). If classifier-independent guidance is enabled, unconditional predictions are fused, and then the prediction noise is fed into the scheduler for updating. This process is iterated until the terminal is reached, yielding the terminal reconstruction latent variables. The third latent variable is used as the second video sample and as the initial non-optimal sample. .

[0059] In this embodiment of the application, generating a video sample pair based on a first video sample and a second video sample includes: extracting a first frame video from the first video sample; replacing the first frame video of the second video sample with the first frame video; generating a positive sample based on the first video sample, generating a negative sample based on the second video sample after the first frame video replacement, and generating a video sample pair based on the positive sample and the negative sample.

[0060] It is understood that in this embodiment of the application, the first frame of the first video sample is used to replace the first frame of the second video sample to construct a video sample pair composed of positive and negative samples. This eliminates the irrelevant bias caused by the difference in the first frame between the positive and negative samples, avoids the model from focusing its training attention on the difference in the first frame, and prompts the model to focus on learning the temporal defects in the subsequent frames of the video sequence. This allows the constructed video sample pair to more accurately reflect the temporal artifact differences in the middle segments of the video.

[0061] It should be noted that the first frame of the video is the first frame in the temporal dimension of the video sample; the positive sample is the preferred sample corresponding to the original clean latent variable. Negative samples are the anchored negative samples after the first frame anchoring is completed. .

[0062] Specifically, the first video sample corresponding to the original clean latent variables is defined as a positive sample. The reconstructed second video sample was initially a non-optimal sample. Extract positive samples The first frame of the video, use that frame to replace The first frame of the video yields anchored negative samples. Based on positive samples anchored negative samples Construct video sample pairs. This first-frame anchoring operation can reduce first-frame drift and structural collapse.

[0063] In step S103, video temporal artifacts in the target time period of the video sample pair are corrected, the loss function of the text generation video diffusion model is determined according to the preference optimization objective of the video diffusion model, and the text generation video diffusion model is trained based on the corrected video samples and the loss function.

[0064] It is understood that the embodiments of this application accurately locate the target time period with high error in the video and specifically correct the time artifacts, realize the centralized allocation of time-series optimization resources, and avoid the gradient dilution problem caused by global uniform supervision. At the same time, by guiding the model iteration through the preference optimization loss function, the problem of discontinuous time sequence and local image distortion in the generated video is effectively improved. While retaining the original effective generated content, the temporal coherence and visual realism of the video generation are greatly improved.

[0065] It should be noted that the target period is the continuous time window where errors are highest and timing defects are most likely to occur. Video temporal artifacts are temporal defects such as motion breakdowns, screen flickering, and temporal discontinuities that occur within a video sequence; the video diffusion model is the currently trainable policy model to be trained and optimized. The preference optimization objective is a centralized implicit preference optimization objective used to guide the model to learn video generation preferences.

[0066] Specifically, step S103 includes two main steps: temporal focus selection based on latent space reconstruction error and calculation of the centralized implicit preference optimization objective. Temporal focus selection based on latent space reconstruction error targets the localization characteristics of video temporal artifacts, focusing on optimizing the time periods most in need of correction. Through frame-level error calculation, sliding window aggregation, selection of the highest error window, and sample truncation, only frames within the high-error window are retained to obtain a centralized positive and negative sample segment. This concentrates the optimization gradient on consecutive frames with significant errors, avoiding wasting optimization resources on normal frames. Unlike existing video DPO (Direct Preference Optimization) and DenseDPO (Fine-grained Temporal Dense Preference Optimization) methods that uniformly weight all frames, this application relies on frame-level error → window aggregation → maximization selection to achieve a strong focusing mechanism that concentrates on the worst consecutive windows.

[0067] In addition to frame-level L2 (L2 squared error) latent space squared error, the embodiments of this application can also use perceptual error, DINO (Distillation with No Labels) / CLIP (Contrastive Language-Image Pre-training) feature space L2 error, and temporal gradient error. In addition to selecting a single maximum average error window, the window selection criteria can be to select the top-M (ranked M) windows for merging / weighting, filter continuous segments based on error thresholds, and dynamically adjust the number and length of windows. Alternatively, the top-k (ranked k) high-error discrete frames can be selected as an alternative (this method may disrupt the short-range temporal structure). All of the above schemes follow the core idea of ​​temporal artifact localization and optimization concentrated on high-error periods.

[0068] After sorting all video frames from highest to lowest according to the average inter-frame error, an overall decreasing error change curve can be obtained: the video frames at the top of the sort have higher error values ​​and are difficult to generate due to their complex details and motion changes. These time-series error-prone frames are few in number and scattered throughout the entire video sequence; the video frames at the bottom of the sort have gradually lower errors and their content tends to be smooth and stable. When the model generates such images, it is less likely to produce distortion or time-series breakage problems.

[0069] In related technologies, preference optimization algorithms apply equal training supervision to all frames of a video, consuming a large amount of computational power on easily generated, stable frames. This results in insufficient optimization for a few high-error, difficult-to-generate frames, making it difficult to thoroughly correct local temporal defects. This application's solution identifies these sparsely distributed high-error segments and concentrates the training gradients onto the continuous temporal windows with the highest errors. It focuses on optimizing the intervals of frames that are difficult to generate and prone to temporal defects, saving redundant computational resources while effectively improving the overall temporal coherence of the video. Figure 2 As shown in step 2, high-error difficult frames are identified through latent space MSE (Mean Squared Error) calculation; for example... Figure 2 As shown in the complete three-stage pipeline diagram, this application sequentially performs preference pair construction, timing error calculation, and centralized window preference optimization iterative update of the model.

[0070] In this embodiment of the application, before correcting the video time artifacts of the target time period in the video sample pair, the method further includes: extracting the first latent variable of the positive sample and the third latent variable of the negative sample in the video sample pair; calculating the squared error of the video sample pair frame by frame based on the first latent variable and the third latent variable; calculating the average difference of each consecutive time window based on the squared error; and taking the time window with the largest average difference as the target time period.

[0071] It is understood that the embodiments of this application calculate the squared error of positive and negative samples frame by frame in the latent space, and use a sliding window to statistically analyze the average difference of each continuous time period. The continuous time window with the highest error is automatically selected as the target time period, which accurately locates the local segments in the video that are prone to temporal artifacts. This provides a clear range for subsequent centralized optimization, prevents optimization resources from being occupied by normal images, and improves the pertinence of temporal defect correction.

[0072] It should be noted that the squared error is the frame-level L2 squared error between the latent variables of positive and negative samples at the same time position; the average difference is the mean of the squared errors of all frames within a single continuous time window; the largest time window is the continuous time interval with the highest average difference among all windows, i.e., the target time period. K is the preset window length, and T is the total number of video frames.

[0073] Specifically, the optimal sample is calculated frame by frame. Anchoring non-preferred samples L2 squared error between latent variables:

[0074] Set a window length K, and calculate the average difference between consecutive time windows by using a sliding window aggregation method:

[0075] Select the index of the window with the largest average difference: Determine the target time window Extract frames within the window to obtain the selected sample fragments after aggregation: Non-optimal sample fragments after aggregation:

[0076] In this embodiment of the application, determining the loss function of the text-to-video diffusion model based on the preference optimization objective of the video diffusion model includes: determining the video sample loss of the video diffusion model based on the preference optimization objective; calculating the model loss of the video diffusion model relative to the reference model based on the video sample loss; calculating the preference loss of the video diffusion model based on the model loss; and generating the loss function of the text-to-video diffusion model based on the preference loss.

[0077] It is understood that the embodiments of this application sequentially calculate the video sample loss, the model loss of the current model relative to the reference model, and the preference loss, and construct a loss function based on the preference loss. The reference model is used as a stable benchmark to measure the model reconstruction deviation, quantify the effect difference between positive and negative samples, form a preference supervision target that is adapted to the time series centralized optimization scenario, and guide the model to iteratively update in the direction of reducing the difference in time series defects.

[0078] It should be noted that the reference model is a text-generated video diffusion model with frozen parameters, used to provide an error baseline; the model loss is the difference in diffusion reconstruction error between the current model and the reference model. The preference loss is a centralized implicit preference optimization loss constructed based on the reconstruction gap between positive and negative samples within a centralized window.

[0079] Specifically, we first define the single-sample loss in the form of diffusion reconstruction error:

[0080] In the cIPO framework, This indicates a trainable video diffusion model. In latent variable samples Text conditions diffusion time step and noise implementation The reconstruction losses below; and They jointly defined from The forward noise addition process before departure, For the model's prediction at this time step, The regression objective is given by the scheduler. Reference model. It is a frozen pre-trained model (and) (Same architecture) to provide a stable performance baseline. Model improvement This reflects the error change of the current model relative to the reference model. It favors logit. Based on this, time-condensed samples were combined. Ultimately, the negative logarithm sigmoid loss is used. Complete centralized implicit preference optimization.

[0081] In addition to the DPO-style negative log-sigmoid form, the preference optimization methods in this application can also employ IPO (Implicit Preference Optimization), SLiC (Self-Like Lihood Correction), RankNet (Rank Network) style pairwise loss, or expand each frame within the concentrated window into multiple sample pairs. Besides the winner-preservation ReLU term, the stabilization regularization term can also use KL (Kullback-Leibler Divergence) divergence constraints, gradient clipping, and temperature coefficient adjustment from the reference model to stabilize training. All of the above methods can widen the gap between positive and negative preferences within the concentrated window and ensure training stability.

[0082] In this embodiment of the application, the expression for the loss function of the preference optimization objective is:

[0083] in, To optimize the loss for centralized implicit preferences, Based on real video and corresponding text conditions Seeking expectations, For the sigmoid function, The expression for a preference for logit is:

[0084] in, , To preset hyperparameters, It is a rectifier linear unit. This represents the difference in diffusion reconstruction error between the video diffusion model and the reference model. To extract positive sample segments after the target time period, This refers to the negative sample segment obtained after extracting the target time period.

[0085] It is understood that the embodiments of this application use centralized implicit preference optimization loss as the training objective, rely on preference logit to quantify the reconstruction error difference between positive and negative sample segments within the target time period, and use ReLU term to set stable regularization constraints to suppress excessive model updates that would damage the original effective generation capability; by combining expectation operation with real video and text conditions to carry out model optimization, the model is guided to reduce the temporal feature differences within the target time period and improve the problem of local video temporal distortion.

[0086] Specifically, the logit preference method uses the difference in reconstruction error between positive and negative sample segments within a set window to measure the quality of a sample. As a winner-preservation stable regularization term, training is finally carried out through centralized implicit preference optimization loss to widen the performance gap between preferred and non-preferred samples within the target time period, thus completing the time-series centralized preference alignment optimization.

[0087] The baseline DPO and DenseDPO generated videos still suffer from local temporal inconsistencies. The cIPO method proposed in this application can effectively improve the temporal coherence of the videos. As shown in the quantitative indicators in Tables 1 and 2, cIPO improves the image fidelity and temporal-related indicators compared to the pre-trained model, DPO, and DenseDPO baselines on the MotionBench (motion video evaluation benchmark dataset) and WISA (general scene video temporal evaluation dataset) test sets. As shown in the ablation experiments in Table 3, constructing negative samples based on noise-added reconstruction and combining them with a temporal concentration strategy can achieve the best results. As shown in the subjective evaluation results in Table 4, cIPO received a higher percentage of votes in the six evaluation dimensions, including human body structure, motion dynamics, and temporal consistency.

[0088] Table 1

[0089] Table 2

[0090] Table 3

[0091] Table 4

[0092] This application conducts a comparative experiment on the number of iteration steps in reverse denoising, selecting multiple sets of different iteration step parameters to comprehensively evaluate the video generation quality from multiple dimensions, including detail restoration, background stabilization, motion smoothness, subject shape preservation, and temporal jitter suppression. Experimental results show that various video quality indicators exhibit a trend of first increasing and then decreasing with adjustments to the number of denoising steps: when the number of denoising steps is insufficient, the model struggles to adequately correct temporal errors, resulting in overall low video generation quality; as the number of steps increases moderately, image detail and temporal coherence improve simultaneously, reaching the optimal range overall; further increasing the number of denoising steps can lead to problems such as distribution shift and detail degradation, causing various performance indicators to gradually decline. Through the above experiments, the optimal denoising parameter values ​​can be determined, achieving a balance between image detail richness and video temporal stability, further ensuring the effective implementation of the temporal preference optimization scheme of this invention.

[0093] This application sets up multiple sets of comparative verification experiments with different denoising step counts. On the one hand, it statistically calculates the error distribution pattern of frame-by-frame reconstruction of video sequences under each set of parameters. On the other hand, it visually compares the video generation effect corresponding to different step counts. An overall consistent trade-off pattern emerges: when the denoising iteration count is low, the video retains richer detail information in each frame, but the temporal fluctuation deviation between frames is greater, resulting in weaker overall temporal coherence. As the denoising step count gradually increases, the inter-frame temporal error continues to converge, the transition between video frames becomes smoother, and the problems of temporal flicker and object displacement are significantly improved. However, excessively high denoising step counts will continuously smooth out image texture details, causing loss of content details. Through a comprehensive consideration of the two sets of comparative results, a compromise and suitable denoising parameter range can be selected, ensuring that the final generated video retains sufficient image detail while maintaining excellent temporal consistency, achieving a balanced and optimal performance between the two.

[0094] The training method for the text-generated video diffusion model proposed in this application utilizes real videos to reconstruct video sample pairs by adding and denoising, and generates supervision signals based on the model's own reconstruction bias. This eliminates the need for manual annotation and external reward models, reducing training costs and external biases. Simultaneously, it locates target time periods with temporal defects and specifically corrects temporal artifacts, concentrating optimization resources on segments prone to defects and avoiding the dilution of training gradients by normal frames. By combining the preference optimization objective with a loss function to conduct model training, it effectively improves the problems of flickering and temporal breaks in generated video images, enhancing the temporal coherence and visual realism of text-generated videos.

[0095] Next, with reference to the accompanying drawings, a training apparatus for a text-generated video diffusion model proposed according to an embodiment of this application is described.

[0096] Figure 3 This is a block diagram of a training device for a text-generated video diffusion model according to an embodiment of this application.

[0097] like Figure 3 As shown, the training device 30 for the text-to-video diffusion model includes: an acquisition module 301, a generation module 302, and a correction module 303.

[0098] The acquisition module 301 is used to acquire the training dataset of the text-to-video diffusion model; the generation module 302 is used to generate a first video sample based on the real video in the training dataset, obtain a second video sample after adding and removing noise from the first video sample, and generate a video sample pair based on the first video sample and the second video sample; the correction module 303 is used to correct the video time artifacts of the target time period in the video sample pair, determine the loss function of the text-to-video diffusion model based on the preference optimization objective of the video diffusion model, and train the text-to-video diffusion model based on the corrected video samples and the loss function.

[0099] In this embodiment, the generation module 302 is further configured to: input real videos from the training dataset into a pre-trained video encoder; generate a first latent variable through the video encoder; and use the first latent variable as a first video sample.

[0100] In this embodiment of the application, the generation module 302 is further configured to: apply forward diffusion to the first video sample to obtain a second latent variable; perform back diffusion for the remaining steps starting from the second latent variable to obtain a third latent variable; and use the third latent variable as the second video sample.

[0101] In this embodiment of the application, the generation module 302 is further configured to: extract the first frame video of the first video sample; replace the first frame video of the second video sample with the first frame video; generate a positive sample based on the first video sample, generate a negative sample based on the second video sample after the first frame video replacement is completed, and generate a video sample pair based on the positive sample and the negative sample.

[0102] In this embodiment of the application, the apparatus 30 further includes a computing module.

[0103] The calculation module is used to extract the first latent variable of the positive samples and the third latent variable of the negative samples in the video sample pair before correcting the video time artifacts in the target time period in the video sample pair; calculate the squared error of the video sample pair frame by frame based on the first latent variable and the third latent variable; calculate the average difference of each consecutive time window based on the squared error; and take the time window with the largest average difference as the target time period.

[0104] In this embodiment, the correction module 303 is further configured to: determine the video sample loss of the video diffusion model based on the preference optimization objective; calculate the model loss of the video diffusion model relative to the reference model based on the video sample loss; calculate the preference loss of the video diffusion model based on the model loss; and generate the loss function of the video diffusion model based on the preference loss.

[0105] In this embodiment of the application, the expression for the loss function of the preference optimization objective is:

[0106] in, To optimize the loss for centralized implicit preferences, Based on real video and corresponding text conditions Seeking expectations, For the sigmoid function, Prefers logit; The expression for preferring logit is:

[0107] in, , To preset hyperparameters, It is a rectifier linear unit. This represents the difference in diffusion reconstruction error between the video diffusion model and the reference model. To extract positive sample segments after the target time period, This refers to the negative sample segment obtained after extracting the target time period.

[0108] It should be noted that the foregoing explanation of the training method embodiment for the text-generated video diffusion model also applies to the training device for the text-generated video diffusion model in this embodiment, and will not be repeated here.

[0109] The training device for the text-generated video diffusion model proposed in this application uses real videos to add and denoise to reconstruct video sample pairs. It generates supervision signals based on the model's own reconstruction bias, eliminating the need for manual annotation and external reward models, thus reducing training costs and external biases. Simultaneously, it locates target time periods with temporal defects and specifically corrects temporal artifacts, concentrating optimization resources on segments prone to defects and avoiding diluting training gradients from normal frames. It combines a loss function constructed with a preference optimization objective to conduct model training, effectively improving the problems of flickering and temporal breaks in the generated video, and enhancing the temporal coherence and visual realism of the text-generated video.

[0110] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include: The memory 401, the processor 402, and the computer program stored on the memory 401 and capable of running on the processor 402.

[0111] When the processor 402 executes the program, it implements the training method for the text-generated video diffusion model provided in the above embodiments.

[0112] Furthermore, electronic devices also include: Communication interface 403 is used for communication between memory 401 and processor 402.

[0113] The memory 401 is used to store computer programs that can run on the processor 402.

[0114] The memory 401 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.

[0115] If the memory 401, processor 402, and communication interface 403 are implemented independently, then the communication interface 403, memory 401, and processor 402 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0116] Optionally, in a specific implementation, if the memory 401, processor 402, and communication interface 403 are integrated on a single chip, then the memory 401, processor 402, and communication interface 403 can communicate with each other through an internal interface.

[0117] Processor 402 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of this application.

[0118] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described training method for the text-to-video diffusion model.

[0119] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0120] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0121] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0122] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any of the following techniques known in the art, or a combination thereof: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.

[0123] Those skilled in the art will understand that all or part of the steps of the methods implementing the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0124] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A training method for a text-to-video diffusion model, characterized in that, Includes the following steps: Obtain the training dataset for the text-to-video diffusion model; A first video sample is generated based on real videos in the training dataset. The first video sample is then subjected to noise addition and removal to obtain a second video sample. Video sample pairs are generated based on the first video sample and the second video sample. Correct the video temporal artifacts in the target time period of the video sample pair, determine the loss function of the text-generated video diffusion model based on the preference optimization objective of the video diffusion model, and train the text-generated video diffusion model based on the corrected video samples and the loss function.

2. The training method for the text-generated video diffusion model according to claim 1, characterized in that, The step of generating the first video sample based on the real videos in the training dataset includes: Input the real videos from the training dataset into the pre-trained video encoder; The video encoder generates a first latent variable, which is then used as the first video sample.

3. The training method for the text-generated video diffusion model according to claim 1 or 2, characterized in that, The step of obtaining the second video sample by adding and removing noise from the first video sample includes: A second latent variable is obtained by applying forward diffusion to the first video sample; Starting from the second latent variable, perform the remaining steps of back diffusion to obtain the third latent variable; The third latent variable is used as the second video sample.

4. The training method for the text-generated video diffusion model according to claim 1, characterized in that, The step of generating video sample pairs based on the first video sample and the second video sample includes: Extract the first frame of the first video sample; The first frame of the second video sample is replaced with the first frame of the video. A positive sample is generated based on the first video sample, a negative sample is generated based on the second video sample after the first frame video replacement is completed, and the video sample pair is generated based on the positive sample and the negative sample.

5. The training method for the text-generated video diffusion model according to claim 1, characterized in that, Before correcting the video temporal artifacts of the target time period in the video sample pair, the method further includes: Extract the first latent variable of the positive samples and the third latent variable of the negative samples from the video sample pairs; The squared error of the video sample pair is calculated frame by frame based on the first latent variable and the third latent variable, and the average difference of each consecutive time window is calculated based on the squared error. The time window with the largest average difference is taken as the target time period.

6. The training method for the text-generated video diffusion model according to claim 1, characterized in that, The step of determining the loss function of the text-generated video diffusion model based on the preference optimization objective of the video diffusion model includes: The video sample loss of the video diffusion model is determined based on the preference optimization objective. The model loss of the video diffusion model relative to the reference model is calculated based on the video sample loss, and the preference loss of the video diffusion model is calculated based on the model loss. The loss function of the text-generated video diffusion model is generated based on the preference loss.

7. The training method for the text-generated video diffusion model according to claim 1 or 6, characterized in that, The expression for the loss function of the preference optimization objective is: in, To optimize the loss for centralized implicit preferences, Based on real video and corresponding text conditions Seeking expectations, For the sigmoid function, For those who prefer logit; The expression for the preference logit is: in, , To preset hyperparameters, It is a rectifier linear unit. This represents the difference in diffusion reconstruction error between the video diffusion model and the reference model. This is to extract the positive sample segment after capturing the target time period. This refers to the negative sample segment obtained after extracting the target time period.

8. A training device for a text-to-video diffusion model, characterized in that, include: The acquisition module is used to acquire the training dataset for the text-to-video diffusion model; The generation module is used to generate a first video sample based on real videos in the training dataset, obtain a second video sample by adding and removing noise from the first video sample, and generate video sample pairs based on the first video sample and the second video sample. The correction module is used to correct video temporal artifacts in the target time period of the video sample pair, determine the loss function of the text-generated video diffusion model according to the preference optimization objective of the video diffusion model, and train the text-generated video diffusion model based on the corrected video samples and the loss function.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the training method for the text-generated video diffusion model according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed, they implement the training method for the text-generated video diffusion model according to any one of claims 1-7.

Images