Video generation model training method and device, electronic equipment and storage medium

Abstract

The application discloses a training method and device of a video generation model, electronic equipment and a storage medium. The method comprises the following steps: calculating the pixel-level motion vector of the obtained training video sequence, and generating a real space-time motion mask for representing a dynamic foreground region and a static background region according to the motion vector; mapping the training video sequence into an initial latent feature and injecting Gaussian noise at a random time step to obtain a noisy latent feature; inputting the noisy latent feature into a denoising network, and respectively outputting predicted noise and a predicted space-time motion mask through a main prediction head and an auxiliary prediction head; based on the real space-time motion mask, calculating a noise reconstruction loss in the dynamic foreground region, a time series variance loss of the denoised feature in the static background region, and an alignment loss of the mask, and updating the parameters of the denoising network based on the above losses. Through dynamic and static region decoupling and asymmetric loss constraint, the application can effectively eliminate the background flicker phenomenon of the generated video, and significantly improve the time sequence coherence and the generation quality.

Description

Technical Field

[0001] This invention relates to the field of video generation technology, specifically to a training method, apparatus, electronic device, and storage medium for a video generation model. Background Technology

[0002] With the continuous advancement of artificial intelligence and computer vision technologies, video generation technology has gradually shifted from traditional rule-based rendering to deep learning-based diffusion models. Existing video generation models typically use variational autoencoders (VAEs) to compress the pixel space of the video into a low-dimensional latent space. During the training phase, Gaussian noise is injected into the continuous latent features of the video through a forward noise addition process. This process then trains a denoising network to predict the added noise, enabling the model to generate continuous video frames based on text or image conditions.

[0003] However, in the actual physical world and video scene generation process, the composition and evolution of images often exhibit spatial heterogeneity, temporal coherence, and high complexity. A single global undifferentiated denoising mechanism is insufficient to independently complete high-quality video reconstruction, and differentiated constraints are usually required for regions with different physical properties. Especially in scenes containing complex actions and vast environments, there are often overlapping features or unclear motion boundaries between dynamic foreground and static background. In such cases, even if the denoising network uses a large 3D attention mechanism for thorough fitting, problems such as flickering background water ripples, sticky subject boundaries, or texture distortion may occur due to the lack of physical semantic decoupling. These problems are usually difficult to expose and correct by increasing the number of network layers or simply expanding the amount of training data, and are not easily detected in traditional global reconstruction error assessments (such as overall MSE loss evaluation). Often, their visual defects only become apparent after the final high-resolution long video is generated.

[0004] Most existing video generation model training methods construct loss optimization systems using three-dimensional feature tensors across all time and space as basic units, focusing on globally unified denoising of the feature dimensions. They lack systematic decoupling modeling and training for the feature evolution of dynamic and static regions in the image. While some models introduce spatiotemporal attention modules or temporal positional encoding, they typically still present the optimization objective with a globally indiscriminate reconstruction loss, failing to reflect the asymmetric information evolution of static backgrounds and dynamic foregrounds within the same video sequence, and also failing to characterize the physical prior differences that static backgrounds should maintain absolute consistency over time. This leads to computationally expensive denoising networks spending significant resources repeatedly fitting static features that don't need to change, while simultaneously amplifying temporal errors through a cascade.

[0005] In summary, existing technologies are insufficient for effectively training and correcting the risks of background flickering and high-frequency misalignment caused by the aliasing of static and dynamic features and global indiscriminate optimization. This not only results in a significant waste of computing power but also limits the actual performance of video generation models in complex physical scenes and long-term sequential generation tasks, and further improvements are needed. Summary of the Invention

[0006] In order to address at least one of the technical defects mentioned in the background art, the present invention aims to provide a training method, apparatus, electronic device, and storage medium for a video generation model.

[0007] A first aspect of the present invention provides a method for training a video generation model, the method comprising the following steps: Calculate the pixel-level motion vectors of the acquired training video sequence, and generate a real spatiotemporal motion mask to represent the dynamic foreground region and static background region in the training video sequence. The training video sequence is input into the encoder and mapped to initial latent features. Gaussian noise is injected into the initial latent features at randomly sampled time steps to obtain noisy latent features corresponding to that time step. The noisy latent features are input into the denoising network to be trained. The backbone prediction head of the denoising network outputs the prediction noise corresponding to the time step, and the auxiliary prediction head outputs the corresponding prediction spatiotemporal motion mask. Based on the real spatiotemporal motion mask, the reconstruction loss of predicted noise and Gaussian noise in the dynamic foreground region and the temporal variance loss of denoised features derived from noisy latent features and predicted noise in the static background region are calculated. The alignment loss between the predicted spatiotemporal motion mask and the real spatiotemporal motion mask is also calculated. The parameters of the denoising network are updated based on the above losses.

[0008] A second aspect of the present invention provides a training apparatus for a video generation model, the apparatus comprising: The mask generation module is used to calculate the pixel-level motion vectors of the acquired training video sequence, and generate a real spatiotemporal motion mask based on the pixel-level motion vectors to characterize the dynamic foreground region and the static background region in the training video sequence. The feature noise-adding module is used to map the training video sequence into the encoder as initial latent features, and inject Gaussian noise into the initial latent features at randomly sampled time steps to obtain noisy latent features corresponding to that time step; The feature prediction module is used to input the noisy latent features into the denoising network to be trained, output the prediction noise corresponding to the time step through the backbone prediction head of the denoising network, and output the corresponding prediction spatiotemporal motion mask through the auxiliary prediction head. The network update module is used to calculate the reconstruction loss of the predicted noise and the Gaussian noise in the dynamic foreground region and the temporal variance loss of the denoised features derived from the noisy latent features and the predicted noise in the static background region based on the real spatiotemporal motion mask, and to calculate the alignment loss between the predicted spatiotemporal motion mask and the real spatiotemporal motion mask, and update the parameters of the denoising network based on the above losses.

[0009] A third aspect of the present invention provides an electronic device, the electronic device including a processor and a memory; the memory is used to store a program; the processor is used to execute the program and, when executing the program, implement the method as described in any of the preceding claims.

[0010] A fourth aspect of the present invention provides a storage medium storing one or more programs that can be executed by one or more processors to implement the method as described in any of the preceding claims.

[0011] This invention introduces a real-time spatiotemporal motion mask during the training phase, decoupling the traditional globally indiscriminate noise addition and denoising process into reconstruction constraints for dynamic foregrounds and temporal variance constraints for static backgrounds. This physically and semantically eliminates the disordered accumulation of denoising errors along the time axis, effectively solving the background flickering and local distortion problems commonly found in existing video diffusion models, thereby improving the temporal coherence and physical realism of the generated video. Simultaneously, combined with a spatially heterogeneous noise addition and mask alignment mechanism, it guides the model to precisely focus its core network capacity on fitting high-frequency complex actions, significantly enhancing the model's generation quality for dynamic subject edge details and overall training efficiency. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the network structure of the video generation model involved in the embodiments of the present invention; Figure 2 This is a flowchart illustrating a training method for a video generation model disclosed in an embodiment of the present invention. Figure 3 This is a schematic diagram of the structure of a training device for a video generation model disclosed in an embodiment of the present invention. Detailed Implementation

[0013] The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit this disclosure.

[0014] This invention provides a method for training a video generation model. It should be noted that the training method for the video generation model described in this embodiment can be deployed on a server cluster or cloud computing platform with high-performance computing capabilities for training the video generation model. This method is particularly suitable for video generation tasks involving complex moving foregrounds and static backgrounds, such as autonomous driving simulation, virtual reality content creation, and security monitoring video generation.

[0015] like Figure 1 As shown, the video generation model used in this embodiment is based on a diffusion architecture and consists of three parts: an encoder, a denoising network, and a decoder. The encoder uses the encoding part of a pre-trained variational autoencoder (VAE) to compress the video sequence in the original pixel space to a low-dimensional latent space and output initial latent features. The decoder is the corresponding VAE decoding part, used to reconstruct the latent features output by the denoising network into the final pixel-level video frames. During training, the parameters of the encoder and decoder remain frozen and do not participate in gradient updates. The denoising network is the core training object of this invention, and it can adopt a 3D U-Net or DiT (Diffusion Transformer) architecture to perform noise prediction on the noisy features in the latent space. In addition, in order to achieve decoupled training between static and dynamic environments, the denoising network also has a dual-branch structure at its end: the main prediction head outputs predicted noise, and the auxiliary prediction head outputs predicted spatiotemporal motion masks.

[0016] Unlike traditional diffusion models that treat video tensors as uniform physical spaces, this method introduces physical semantic-level dynamic and static space priors to bidirectionally modify the forward noise addition and reverse noise reduction processes, thereby solving problems such as background flickering and motion boundary blurring caused by global indiscriminate optimization in existing technologies.

[0017] like Figure 2 As shown, the method includes the following steps: S1, calculate the pixel-level motion vector of the acquired training video sequence, and generate a real spatiotemporal motion mask based on the pixel-level motion vector to characterize the dynamic foreground region and static background region in the training video sequence. In the initial stage of training, a training video sequence is first acquired, and an optical flow estimation algorithm (such as the RAFT algorithm) is used to calculate the pixel-level motion vectors of adjacent frames in the video sequence. It should be understood that the motion vector reflects the displacement direction and magnitude of each pixel between adjacent frames and can be used to distinguish between dynamic foreground and static background in subsequent training.

[0018] However, directly dividing the region based solely on the original motion vectors will produce jagged noise at the object edges due to optical flow calculations, and the original resolution will be inconsistent with the resolution of subsequent latent space features. Therefore, further processing is required to generate a high-precision, realistic spatiotemporal motion mask. Specifically: As an example, generating a real spatiotemporal motion mask based on the pixel-level motion vectors to characterize dynamic foreground regions and static background regions in a training video sequence includes: S11, calculate the amplitude of the pixel-level motion vector, and compare the amplitude with a preset motion judgment threshold pixel by pixel. Mark the pixels with amplitudes greater than or equal to the motion judgment threshold as the dynamic foreground region, and mark the pixels with amplitudes less than the motion judgment threshold as the static background region. Specifically, for each pixel, its motion vector includes a horizontal component. and vertical component The amplitude is The amplitude is compared pixel by pixel with a preset motion detection threshold, which can be set according to the noise level and motion sensitivity of the actual scene, for example, 1.5 pixels.

[0019] If the motion amplitude of a pixel is greater than or equal to the motion judgment threshold mentioned above, the pixel is considered to belong to a dynamic foreground area (such as moving people, vehicles, etc.); if it is less than the motion judgment threshold, it is considered to belong to a static background area (such as sky, walls, road surface, etc.).

[0020] By comparing each pixel as described above, consecutive video frames are mapped to an initial binary motion mask, where the foreground is marked as 1 and the background as 0.

[0021] S12, obtain the real spatiotemporal motion mask based on the labeling results.

[0022] After obtaining the initial binarized motion mask, directly using this mask for subsequent training presents the following problems: isolated noise or small holes at the object edges due to optical flow calculation errors; and the spatial resolution of the mask is inconsistent with the initial latent feature resolution output by the subsequent encoder, making it unsuitable for direct pixel-level constraints. Therefore, this embodiment further optimizes the labeling results.

[0023] As an example, the real spatiotemporal motion mask obtained based on the labeling results includes: The marking result is used as an initial binary motion mask, and morphological smoothing filtering is applied to the initial binary motion mask to eliminate noise at the boundary between the dynamic foreground region and the static background region. Specifically, morphological smoothing filtering employs an opening operation that first erodes and then dilates, and the kernel size can be flexibly set according to the video resolution and object scale. Preferably, a 3×3 pixel rectangular structuring element can be used as the convolution kernel.

[0024] The erosion operation works as follows: It iterates through each pixel in the initial binary motion mask, extracts all pixel values ​​within a 3×3 neighborhood centered on that pixel, and takes the minimum value within the neighborhood as the new value for that pixel. Since the background area is typically 0 and the foreground area is 1, any isolated white foreground noise (i.e., scattered background pixels mistakenly labeled as foreground) will have most pixels with values ​​of 0 within the 3×3 neighborhood. After the minimum value operation, these values ​​will be reset to 0, effectively removing isolated noise outside the foreground area. Similarly, the dilation operation takes the maximum value within the neighborhood, which can fill the small holes inside the foreground area caused by optical flow calculation occlusion or sparse texture, making the foreground area more complete and coherent.

[0025] It should be understood that the opening operation's order of erosion followed by dilation ensures that while removing external noise, the boundaries of the foreground region are not excessively expanded. This results in smoother, more continuous boundaries between moving and static areas, better reflecting the true contours of objects in the physical world. Additionally, for scenes where edges still exhibit jaggedness, an additional closing operation (dilation followed by erosion) can be performed to further smooth the edges. However, care must be taken to avoid over-smoothing that could cause the moving area to shrink.

[0026] A spatial downsampling operation is performed on the filtered binarized motion mask to match its spatial resolution with that of the initial latent feature, thus obtaining the real spatiotemporal motion mask.

[0027] Since the original video sequence typically has a high spatial resolution (e.g., 256×256 or 512×512), while the initial latent features output by the VAE encoder have a significantly reduced resolution after multiple convolutional downsampling (e.g., 32×32 or 64×64), there exists a fixed downsampling ratio between the two. To achieve accurate separation of motion and static regions in the latent space, the resolution of the motion mask must be aligned with the latent features. The downsampling operation can employ nearest-neighbor interpolation or bilinear interpolation followed by binarization.

[0028] Specifically, if nearest neighbor interpolation is used, for each pixel position in the downsampled target mask, the pixel value at the center of the corresponding receptive field in the original mask is directly selected as the sampling result. This method is computationally efficient and can maintain the binary properties of the original mask. If bilinear interpolation is used, the filtered mask is first scaled by floating-point interpolation, and then re-binarized with a threshold of 0.5 to generate the final binary real spatiotemporal motion mask.

[0029] It should be understood that, regardless of the interpolation method used, the size of the mask space after downsampling must be strictly consistent with the width and height of the initial latent feature. For example, when the latent feature resolution is 32×32, the original 256×256 mask is reduced by a downsampling factor of 8.

[0030] After the above morphological smoothing filtering and spatial downsampling processing, a high-precision real spatiotemporal motion mask is finally obtained, denoted as . In this model, the pixel values ​​of the dynamic foreground region are marked as 1, and the pixel values ​​of the static background region are marked as 0. It should be understood that this real spatiotemporal motion mask not only eliminates edge noise caused by optical flow calculation, but also aligns with the latent feature space, thus serving as a physical prior for subsequent heterogeneous noise addition and asymmetric loss calculation.

[0031] S2, the training video sequence is input to the encoder and mapped to initial latent features, and Gaussian noise is injected into the initial latent features at randomly sampled time steps to obtain noisy latent features corresponding to that time step; This step utilizes a pre-trained variational autoencoder (VAE) to compress the original training video sequence from pixel space to a low-dimensional latent space, obtaining initial latent features. This encoder typically consists of multiple stacked 3D convolutional layers and downsampling layers, progressively reducing the spatial resolution to 1 / 8 or 1 / 16 of the original size while expanding the number of channels to the latent feature dimension (e.g., 4 or 8 channels). For example, the input video sequence is... Then the encoder outputs the initial latent features The shape is ,in , , This is the total downsampling factor. This represents the number of latent feature channels.

[0032] In the forward noise addition process of the diffusion model, a time step is randomly sampled from a preset time step range (e.g., 1 to 1000). Gaussian noise is injected into the initial latent feature to obtain the noisy latent feature corresponding to that time step. Traditional diffusion models employ a globally uniform Gaussian noise scheduling strategy at any time step, applying Gaussian noise with the same variance to all locations across the entire feature map. However, dynamic foregrounds contain numerous high-frequency spatiotemporal features (such as edges, textures, and rapid motion), while static backgrounds dominate low-frequency structures (such as flat regions and slow changes). Uniform noise application leads to signal-to-noise ratio imbalance and feature annihilation of high-frequency dynamic features early in the noise application process, making it difficult for the model to learn fine motion boundaries. Meanwhile, insufficient noise is applied to the static background, hindering effective training of denoising capabilities.

[0033] To address the aforementioned technical problems, this embodiment designs and employs a mask-based spatial heterogeneous noise addition strategy, as detailed below: As an example, injecting Gaussian noise into the initial latent feature at a randomly sampled time step to obtain a noisy latent feature corresponding to that time step includes: S21, obtain the baseline noise variance scheduling parameter corresponding to the time step, and decouple the baseline noise variance scheduling parameter into a first noise variance corresponding to the dynamic foreground region and a second noise variance corresponding to the static background region based on the real spatiotemporal motion mask; wherein, the rate of change of the first noise variance on the time axis is different from that of the second noise variance. Among them, the baseline noise variance scheduling parameter The definition is: in the forward noise addition process of the diffusion model, a set of time steps is pre-set. Monotonically increasing noise variance coefficient ,in This represents the total number of time steps (e.g., 1000). Controlled in the first The intensity of Gaussian noise injected into the step-to-latent feature. The larger the value, the stronger the added noise. Common scheduling methods include linear scheduling and cosine scheduling.

[0034] For example, linear scheduling can be represented as ,in , Cosine tuning can be expressed as ,in , .

[0035] Specifically, at the current sampling time step First, obtain the baseline noise variance scheduling parameters corresponding to that time step. (That is, take the first one from the above scheduling sequence) (a number of values). However, unlike traditional methods that directly unify this globally... Applied to all pixels, this embodiment employs a mask-based decoupling strategy to decouple a single pixel. The noise variance parameters are transformed into two spatially heterogeneous sets, which are configured independently for the dynamic foreground region and the static background region, respectively.

[0036] The specific implementation of decoupling is as follows: Two completely different noise variance scheduling curves are pre-constructed, serving the dynamic foreground region and the static background region respectively. The first noise variance scheduling curve... Employing a scheduling method with a high rate of change, such as the aforementioned cosine scheduling, which rises rapidly on the time axis, allows the dynamic foreground region to experience higher noise intensity at earlier time steps; the second noise variance scheduling curve Employing a scheduling method with a low rate of change, such as linear scheduling, which grows slowly over time, ensures that the static background region remains at a low noise level.

[0037] During training, for the current sampling time step The variance values ​​were obtained from the two independent scheduling curves mentioned above. and Then, using real-time motion masks. Perform pixel-by-pixel selection; specifically: for positions with a mask value of 1 (dynamic foreground), select... For positions with a mask value of 0 (static background), select... This method decouples the original single baseline noise variance scheduling parameter into two spatially heterogeneous sets of variance parameters. Because the rates of change of the two curves on the time axis are forced to be different, the first noise variance and the second noise variance exhibit significant differences.

[0038] As another simplified implementation, a global baseline can also be obtained first. (For example, using linear scheduling), and then modulating it based on the mask, specifically: multiplying the foreground region by a scaling factor greater than 1. Multiply the background area by a scaling factor less than 1. ,Right now , This also achieves the effect of different rates of change between the first and second noise variances. Among them, , Furthermore, the specific values ​​of both can be preset or dynamically adjusted according to the training task.

[0039] Through the decoupling operations described above, this embodiment enables the dynamic foreground region and the static background region to have their own independent noise evolution trajectories within the same video sequence. Furthermore, the rates of change of the first noise variance and the second noise variance are forcibly set to be different, thereby achieving a noise addition strategy that adapts to the physical information density. This design effectively avoids the coupling problem of premature annihilation of high-frequency foreground features and insufficient noise addition to the low-frequency background in traditional unified noise addition.

[0040] S22, sample according to the first noise variance and the second noise variance respectively to generate global Gaussian noise with spatial heterogeneous variance characteristics, and inject the global Gaussian noise into the initial latent feature so that the dynamic foreground region and the static background region present asymmetrical noise intensity, thereby obtaining the noisy latent feature.

[0041] After obtaining the two independent variance scheduling branches mentioned above, noise is sampled from the standard normal distribution, and scaled according to the variance of each branch. Specifically, for the dynamic foreground region, from... Medium sampling noise tensor For static background areas, from Medium sampling noise tensor Among them, the spatial dimensions of these two noise tensors are related to the initial latent features. Exactly the same.

[0042] Using real-time motion masks Perform pixel-by-pixel noise concatenation; specifically: at positions where the mask value is 1, take... The corresponding pixel value; at the position where the mask value is 0, take The corresponding pixel values ​​are then used to form a global Gaussian noise tensor with spatial heterogeneous variance characteristics. That is, the noise variance is different at different spatial locations, and it strictly follows the physical division of dynamic and static areas.

[0043] Injecting this global Gaussian noise into the initial latent features yields the noisy latent features: ,in, These are the cumulative noise scheduling parameters.

[0044] It should be understood that, due to Different regions have different variance characteristics, resulting in noisy latent features. The dynamic foreground region exhibits a higher noise intensity (features are severely damaged by noise), while the static background region exhibits a lower noise intensity (features are relatively clear). This asymmetric noise intensity distribution allows the subsequent denoising network to learn selectively during the reverse process: the foreground needs to recover high-frequency details, while the background only needs to maintain low-frequency stability, thereby significantly improving training efficiency and generation quality.

[0045] S3, the noisy latent features are input into the denoising network to be trained, the backbone prediction head of the denoising network outputs the prediction noise corresponding to the time step, and the auxiliary prediction head outputs the corresponding prediction spatiotemporal motion mask. After completing forward noise addition and obtaining the current time step Noise-adding latent features This step then feeds the input into the denoising network to be trained for forward inference. The denoising network can employ a typical diffusion-based architecture, such as 3DU-Net or DiT (Diffusion Transformer), whose input includes the noisy latent features. In addition, it may include the current time step. The encoding (such as sinusoidal positional encoding) and optional text conditional vectors (if conditional video generation is performed). It should be understood that the core function of this denoising network is to predict the position of the target video at time step [missing information]. The noise added during the process is gradually removed to restore clear video features during the reverse process.

[0046] To achieve the decoupled training of dynamic and static regions proposed in this invention, this embodiment employs a dual-branch design at the output of the denoising network. Specifically, the denoising network splits into two parallel prediction heads after the last feature map layer: Backbone prediction head: This prediction head consists of one or more convolutional layers, with the number of output channels equal to the input noisy latent features. The number of channels is the same. Its output is denoted as... This refers to the prediction noise corresponding to that time step.

[0047] It should be understood that this predicted noise is the Gaussian noise that the model believes was added during the noise addition process, and its spatial size is similar to... Completely consistent. In traditional diffusion models, only the predicted noise needs to be output to calculate the reconstruction loss, while in this embodiment, the backbone prediction head is responsible for providing noise reconstruction error for the subsequent dynamic foreground region.

[0048] Auxiliary Prediction Head: This prediction head is a lightweight sub-network, for example, composed of two stacked 3D convolutional layers (the first layer has a 3×3×3 kernel size, and the second layer has a 1×1×1 kernel size), with the last layer connected to a sigmoid activation function. The output of this prediction head has 1 channel, representing the probability that each spatial location belongs to a dynamic foreground region. Its output is denoted as... That is, the prediction of spatiotemporal motion mask.

[0049] The spatial resolution and noisy latent features of the predicted spatiotemporal motion mask Same (e.g.) Each pixel value, ranging from 0 to 1, represents the model's confidence in whether that location belongs to the dynamic foreground. It should be understood that by using an auxiliary prediction head, the denoising network implicitly learns to distinguish between moving and stationary regions in the image while learning to remove noise, thus providing adaptive masking information for subsequent asymmetric loss calculations.

[0050] In practice, the main prediction head and the auxiliary prediction head share the backbone of the denoising network (i.e., all downsampling, intermediate bottleneck, and upsampling layers), branching only at the last feature map layer. This design maintains efficient use of network capacity while allowing the two prediction tasks to complement each other. Accurate noise prediction requires understanding motion boundaries, while accurate motion mask prediction helps the denoising network apply different processing strategies to the foreground and background.

[0051] S4. Based on the real spatiotemporal motion mask, calculate the reconstruction loss of predicted noise and Gaussian noise in the dynamic foreground region, and the temporal variance loss of denoised features derived from noisy latent features and predicted noise in the static background region. Also calculate the alignment loss between the predicted spatiotemporal motion mask and the real spatiotemporal motion mask, and update the parameters of the denoising network based on the above losses.

[0052] Existing diffusion models generally use global mean squared error (MSE) as the sole optimization objective, neglecting the physical prior that the static background should remain absolutely consistent over time. This leads to severe water ripples or background flickering in the generated video in background areas (such as white walls and skies). To eliminate this defect, this embodiment implements three targeted asymmetric loss evaluations based on real-time motion masks, corresponding to dynamic foreground regions, static background regions, and mask prediction tasks, respectively. These evaluations are then weighted and summed to update the parameters of the denoising network.

[0053] Specifically, the following three losses are calculated simultaneously: Reconstruction losses This is used to constrain the difference between predicted noise and true Gaussian noise within a dynamic foreground region; temporal variance loss. This is used to constrain the variance of denoised features over time within a static background region, forcing the background to remain constant across frames; alignment loss. It is used to constrain the difference between the predicted spatiotemporal motion mask and the real spatiotemporal motion mask, guiding the model to learn the motion boundary.

[0054] The calculation method and function of each type of loss are explained below.

[0055] (1) Reconstruction losses : Using real-time motion masks Extract the dynamic foreground region (locations with a mask value of 1). Within this dynamic foreground region, calculate the predicted noise output by the denoising network. Compared with injected real Gaussian noise The mean square error between them. Since heterogeneous noise addition is used in step S2, the true Gaussian noise in the dynamic foreground region follows... Therefore, this loss can be used to measure the model's prediction accuracy for high-frequency dynamic features. The formula for the reconstruction loss can be expressed as:

[0056] in, This indicates element-wise multiplication.

[0057] It should be understood that this loss forces the model to precisely focus the core network capacity on the accurate reproduction of complex actions, while not imposing this constraint on static background regions, thus avoiding the model wasting computing power to fit low-frequency components that should not change.

[0058] (2) Time series variance loss : For static background regions, traditional frame-by-frame MSE loss cannot constrain feature fluctuations between frames. Even if the pixel values ​​of each frame are reconstructed independently and precisely, slight inconsistencies between frames (such as brightness fluctuations or texture drift) will result in noticeable flickering during playback. Therefore, this embodiment directly derives the denoising features from the backward process of the diffusion model and calculates its temporal variance loss function, as detailed below: As an example, the temporal variance loss of the denoised features derived from the noisy latent features and the predicted noise within a static background region is calculated as follows: Based on the noise latent features, the predicted noise, and the baseline noise variance scheduling parameters corresponding to the time step, the denoising features corresponding to the current time step are derived using the back diffusion formula. Specifically, in the reverse process of the diffusion model, given the noisy latent features... and prediction noise It is possible to estimate the original latent features. The approximate value, i.e., the denoising feature. Preferably, this embodiment uses the inverse prediction formula commonly used in DDPM (Denoising Diffusion Probabilistic Models):

[0059] in, For cumulative noise scheduling parameters, The baseline noise variance scheduling parameter.

[0060] It should be understood that the above formula is mathematically equivalent to subtracting the contribution of the predicted noise from the current noise features and then dividing by the retention coefficient of the original signal, thus obtaining a point estimate of the original clean latent features. It should be noted that this estimate is not required to be completely accurate; its value varies with time steps. The accuracy will gradually improve; however, during the training process, even with noisy estimates, the variance in the time dimension can still effectively reflect the degree of feature fluctuation in the background region.

[0061] The spatial feature tensor of the denoised features in the static background region is extracted using the real spatiotemporal motion mask, and the feature sequence of the feature sequence is calculated as the time-series variance loss.

[0062] Specifically, the denoising features are obtained Afterwards, its dimension is the same as the initial latent feature, denoted as... ,in For video frame rate, For spatial resolution, This represents the number of channels.

[0063] Using real-time motion masks (After downsampling, the resolution is consistent with the latent feature) Extract the static background region: for each spatial location with a mask value of 0 ,from Extract the feature vector (of length ) at this position from the frame. ), forming a The sequence. To measure the stability of this position on the time axis, the variance of the sequence in the time dimension (the first dimension) is calculated.

[0064] Specifically, each channel is calculated independently first. The variance of each value is averaged over all channels to obtain the temporal variance value for that spatial location. Then, the variance values ​​for all static background spatial locations are averaged to obtain the final temporal variance loss.

[0065] in, This represents the set of all spatial coordinates of the static background region (mask value of 0). This represents the total number of pixels in the region. This represents the number of channels.

[0066] It should be understood that this loss term directly penalizes the fluctuations of background features on the time axis, which can force the denoising network to learn to generate a temporally consistent background representation. Unlike traditional L2 temporal smoothing loss (such as the L2 norm of the difference between adjacent frames), variance loss does not depend on the local difference of the frame order, but directly measures the dispersion of the overall time series, which is more in line with the physical prior that the static background should remain unchanged between any two frames.

[0067] (3) Alignment loss : To guide the auxiliary prediction head in outputting an accurate predicted spatiotemporal motion mask, this embodiment calculates the prediction mask. With the real mask The binary cross-entropy loss is used. Since the true mask is binary (0 or 1), while the predicted mask is a continuous value (0 to 1) after Sigmoid activation, cross-entropy provides a smooth gradient. The loss is defined as:

[0068] The process involves summing up all time steps and spatial locations.

[0069] It should be understood that this loss prompts the denoising network to implicitly learn motion boundaries during the inverse denoising process, thereby improving its ability to distinguish between moving and static regions. When the prediction mask is accurate, the subsequent reconstruction loss and temporal variance loss can also be applied more accurately to the correct regions, forming positive feedback.

[0070] Dynamic weighting coefficients are configured for the three types of losses mentioned above. , , As an example, it can be set to... , , The specific value can be adjusted based on the validation loss during training; no specific limit is imposed. The total loss is obtained by weighted summation:

[0071] The gradient is calculated using the backpropagation algorithm, and the parameters of the denoising network are updated using an optimizer (such as AdamW) until the model converges. During training, the real-time motion mask is only used for loss calculation and does not participate in forward propagation, thus not increasing the burden on the inference stage.

[0072] Through the operation of the above embodiments, the denoising network can not only eliminate background flicker, but also protect high-frequency motion features through heterogeneous noise addition. At the same time, it can improve the perception of motion boundaries by utilizing mask alignment loss, thereby significantly enhancing the temporal coherence and visual quality of long video generation.

[0073] Please see Figure 3 The present invention provides a training device 100 for a video generation model, the device comprising: The mask generation module 10 is used to calculate the pixel-level motion vector of the acquired training video sequence, and generate a real spatiotemporal motion mask based on the pixel-level motion vector to characterize the dynamic foreground region and static background region in the training video sequence. The feature noise-adding module 20 is used to map the training video sequence into the encoder as initial latent features, and inject Gaussian noise into the initial latent features at randomly sampled time steps to obtain noisy latent features corresponding to that time step; The feature prediction module 30 is used to input the noisy latent features into the denoising network to be trained, output the prediction noise corresponding to the time step through the backbone prediction head of the denoising network, and output the corresponding prediction spatiotemporal motion mask through the auxiliary prediction head. The network update module 40 is used to calculate the reconstruction loss of the predicted noise and the Gaussian noise in the dynamic foreground region and the temporal variance loss of the denoised features derived from the noisy latent features and the predicted noise in the static background region based on the real spatiotemporal motion mask, and to calculate the alignment loss between the predicted spatiotemporal motion mask and the real spatiotemporal motion mask, and update the parameters of the denoising network based on the above losses.

[0074] As an example, the mask generation module 10 is configured to: Calculate the amplitude of the pixel-level motion vector, and compare the amplitude with a preset motion judgment threshold pixel by pixel. Mark the pixels with amplitudes greater than or equal to the motion judgment threshold as the dynamic foreground region, and mark the pixels with amplitudes less than the motion judgment threshold as the static background region. The real spatiotemporal motion mask is obtained based on the labeling results.

[0075] As an example, when the mask generation module 10 obtains the real spatiotemporal motion mask based on the marking results, it is specifically configured as follows: The marking result is used as an initial binary motion mask, and morphological smoothing filtering is applied to the initial binary motion mask to eliminate noise at the boundary between the dynamic foreground region and the static background region. A spatial downsampling operation is performed on the filtered binarized motion mask to match its spatial resolution with that of the initial latent feature, thus obtaining the real spatiotemporal motion mask.

[0076] As an example, the feature noise-adding module 20 is configured to: Obtain the baseline noise variance scheduling parameter corresponding to the time step, and decouple the baseline noise variance scheduling parameter into a first noise variance corresponding to the dynamic foreground region and a second noise variance corresponding to the static background region based on the real spatiotemporal motion mask; wherein, the rate of change of the first noise variance on the time axis is different from that of the second noise variance. Sampling is performed according to the first noise variance and the second noise variance respectively to generate global Gaussian noise with spatial heterogeneous variance characteristics. The global Gaussian noise is injected into the initial latent feature to make the dynamic foreground region and the static background region exhibit asymmetrical noise intensity, thereby obtaining the noisy latent feature.

[0077] As an example, the network update module 40 is specifically configured to calculate the temporal variance loss as follows: Based on the noise latent features, the predicted noise, and the baseline noise variance scheduling parameters corresponding to the time step, the denoising features corresponding to the current time step are derived using the back diffusion formula. The spatial feature tensor of the denoised features in the static background area is extracted using the real spatiotemporal motion mask, and the feature sequence of the feature sequence is calculated as the time-series variance loss.

[0078] It should be understood that each module in the training device of the video generation model corresponds one-to-one with each step in the above method embodiment, and its specific implementation and technical effects have been described in detail in the method embodiment, and will not be repeated here.

[0079] This invention also provides an electronic device, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor. The memory stores the program and related data, and the processor executes the program. When the processor executes the program, it can implement the training method for the video generation model as described in any of the preceding embodiments.

[0080] As an example, the processor can be a central processing unit (CPU), graphics processing unit (GPU), neural network processing unit (NPU), tensor processing unit (TPU), digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device. The memory can be read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), solid-state drive (SSD), or other non-volatile storage media. This electronic device can be deployed as a single server, server cluster, cloud server, workstation, or high-performance computing platform, or integrated into, for example, an autonomous driving simulation system, virtual reality content generation device, or security video analysis server. By running the above method, this electronic device can eliminate background flickering when training the video generation model, improving the temporal coherence and physical realism of the generated video.

[0081] This invention also provides a storage medium storing one or more programs (i.e., computer-executable instructions), which can be executed by one or more processors to implement the training method for the video generation model as described in any of the preceding claims.

[0082] The storage medium can be any one or more combinations of a disk, optical disk, USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magneto-optical disk, flash memory, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc. The storage medium can exist independently of the electronic device (e.g., optical disk, USB flash drive) or be integrated inside the electronic device (e.g., built-in solid-state drive). When the one or more programs are executed by a processor, the processor performs all or part of the steps in the above method. By using this storage medium, users or device manufacturers can easily deploy the method of this invention to any compatible computing device to achieve efficient and high-quality video generation model training.

[0083] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A training method for a video generation model, characterized in that, Includes the following steps: Calculate the pixel-level motion vectors of the acquired training video sequence, and generate a real spatiotemporal motion mask based on the pixel-level motion vectors to characterize the dynamic foreground region and static background region in the training video sequence; The training video sequence is input into the encoder and mapped to initial latent features. Gaussian noise is injected into the initial latent features at randomly sampled time steps to obtain noisy latent features corresponding to that time step. The noisy latent features are input into the denoising network to be trained. The backbone prediction head of the denoising network outputs the prediction noise corresponding to the time step, and the auxiliary prediction head outputs the corresponding prediction spatiotemporal motion mask. Based on the real spatiotemporal motion mask, the reconstruction loss of predicted noise and Gaussian noise in the dynamic foreground region and the temporal variance loss of denoised features derived from noisy latent features and predicted noise in the static background region are calculated. The alignment loss between the predicted spatiotemporal motion mask and the real spatiotemporal motion mask is also calculated. The parameters of the denoising network are updated based on the above losses.

2. The training method for a video generation model according to claim 1, characterized in that: Based on the pixel-level motion vectors, a real spatiotemporal motion mask is generated to characterize the dynamic foreground region and static background region in the training video sequence, including: Calculate the amplitude of the pixel-level motion vector, and compare the amplitude with a preset motion judgment threshold pixel by pixel. Mark the pixels with amplitudes greater than or equal to the motion judgment threshold as the dynamic foreground region, and mark the pixels with amplitudes less than the motion judgment threshold as the static background region. The real spatiotemporal motion mask is obtained based on the labeling results.

3. The training method for a video generation model according to claim 1, characterized in that: The real spatiotemporal motion mask is obtained based on the labeling results, including: The labeling result is used as an initial binary motion mask. Morphological smoothing filtering is applied to the initial binary motion mask to eliminate noise at the boundary between the dynamic foreground region and the static background region. Spatial downsampling is performed on the filtered binary motion mask to match its spatial resolution with the spatial resolution of the initial latent feature, thus obtaining the real spatiotemporal motion mask.

4. The training method for a video generation model according to claim 1, characterized in that: Injecting Gaussian noise into the initial latent features at randomly sampled time steps to obtain noisy latent features corresponding to that time step includes: Obtain the baseline noise variance scheduling parameter corresponding to the time step, and decouple the baseline noise variance scheduling parameter into a first noise variance corresponding to the dynamic foreground region and a second noise variance corresponding to the static background region based on the real spatiotemporal motion mask; wherein, the rate of change of the first noise variance on the time axis is different from that of the second noise variance. Sampling is performed according to the first noise variance and the second noise variance respectively to generate global Gaussian noise with spatial heterogeneous variance characteristics. The global Gaussian noise is injected into the initial latent feature to make the dynamic foreground region and the static background region exhibit asymmetrical noise intensity, thereby obtaining the noisy latent feature.

5. The training method for a video generation model according to claim 1, characterized in that: The temporal variance loss of the denoised features derived from the noisy latent features and the predicted noise within the static background region is calculated as follows: Based on the noise latent features, the predicted noise, and the baseline noise variance scheduling parameters corresponding to the time step, the denoising features corresponding to the current time step are derived using the back diffusion formula. The spatial feature tensor of the denoised features in the static background region is extracted using the real spatiotemporal motion mask, and the feature sequence of the feature sequence is calculated as the time-series variance loss.

6. A training device for a video generation model, characterized in that, The device includes: The mask generation module is used to calculate the pixel-level motion vectors of the acquired training video sequence, and generate a real spatiotemporal motion mask based on the pixel-level motion vectors to characterize the dynamic foreground region and the static background region in the training video sequence. The feature noise-adding module is used to map the training video sequence into the encoder as initial latent features, and inject Gaussian noise into the initial latent features at randomly sampled time steps to obtain noisy latent features corresponding to that time step; The feature prediction module is used to input the noisy latent features into the denoising network to be trained, output the prediction noise corresponding to the time step through the backbone prediction head of the denoising network, and output the corresponding prediction spatiotemporal motion mask through the auxiliary prediction head. The network update module is used to calculate the reconstruction loss of the predicted noise and the Gaussian noise in the dynamic foreground region and the temporal variance loss of the denoised features derived from the noisy latent features and the predicted noise in the static background region based on the real spatiotemporal motion mask, and to calculate the alignment loss between the predicted spatiotemporal motion mask and the real spatiotemporal motion mask, and update the parameters of the denoising network based on the above losses.

7. The training apparatus for a video generation model according to claim 6, characterized in that, The feature noise-adding module is configured as follows: Obtain the baseline noise variance scheduling parameter corresponding to the time step, and decouple the baseline noise variance scheduling parameter into a first noise variance corresponding to the dynamic foreground region and a second noise variance corresponding to the static background region based on the real spatiotemporal motion mask; wherein, the rate of change of the first noise variance on the time axis is different from that of the second noise variance. Sampling is performed according to the first noise variance and the second noise variance respectively to generate global Gaussian noise with spatial heterogeneous variance characteristics. The global Gaussian noise is injected into the initial latent feature to make the dynamic foreground region and the static background region exhibit asymmetrical noise intensity, thereby obtaining the noisy latent feature.

8. The training apparatus for a video generation model according to claim 6, characterized in that, The network update module is specifically configured to calculate the temporal variance loss as follows: Based on the noise latent features, the predicted noise, and the baseline noise variance scheduling parameters corresponding to the time step, the denoising features corresponding to the current time step are derived using the back diffusion formula. The spatial feature tensor of the denoised features in the static background area is extracted using the real spatiotemporal motion mask, and the feature sequence of the feature sequence is calculated as the time-series variance loss.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory; the memory is used to store a program; the processor is used to execute the program and, when executing the program, implement the method as described in any one of claims 1 to 5.

10. A storage medium, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the method as described in any one of claims 1 to 5.

Images