A video prediction method and system based on a dynamic diffusion model

Abstract

The present application relates to computer vision technology, aiming at providing a video prediction method and system based on dynamic diffusion model. It includes: a feature extraction network formed by stacking a plurality of serial feature extraction modules, and the activation state is controlled by a routing module; each pair of input frames is split into parallel spatial feature branches and motion feature branches, and after branch separation and scale transformation processing, the fusion features with spatial details and motion correlation are output, and then the last frame is encoded and fused as the conditional information input into the diffusion model; the next moment prediction optical flow field is generated by step-by-step denoising, the position of the current pixel point in the next moment is calculated through forward warping operation, and the final prediction frame is synthesized; the prediction frame is connected with the historical video frame sequence through repeated operation, and the continuous prediction task is realized. The present application can adaptively learn the features of the current and past driving environment, and use the randomness of the diffusion model to generate the prediction optical flow, so as to achieve the purpose of video frame prediction.

Description

Technical Field

[0001] This invention relates to computer vision technology, and in particular to a video prediction method and system based on a dynamic diffusion model. Background Technology

[0002] Autonomous driving agents perceive their surroundings, predict future environmental changes, and make corresponding decisions and actions. This represents a promising yet challenging application in the field of artificial intelligence. The goal of video frame prediction is to generate future frames based on historical frame sequences or single context frames; however, this requires models to have a deeper understanding of dynamic driving scenarios and temporal consistency. Several challenges have long existed in the field of video prediction, including the need for algorithms that balance low-level pixel processing with high-level scene dynamic understanding, the inadequacy of perception and stochastic evaluation metrics, difficulties in achieving long-term synthesis, and the limited availability of high-quality driving scene data.

[0003] STDiff is a spatiotemporal diffusion model for predicting continuous stochastic videos. By separating motion and content information in the video, it can handle dynamic information more accurately. However, the STDiff model architecture is relatively complex, especially when using stochastic differential equations (SDEs) for spatiotemporal modeling and autoregressive diffusion generation. This results in high computational costs, slow training and inference speeds, and requires powerful hardware support. This limits its application scenarios with limited resources.

[0004] DMVFN (Dynamic Multi-Scale Voxel Flow Network) is an efficient deep learning model specifically designed for video prediction tasks. It captures complex motion changes between video frames through dynamic multi-scale voxel flow estimation. However, for scenes with high uncertainty or bifurcation behavior, DMVFN may struggle to accurately reflect all potential future developments over long periods. This model relies on extrapolation from past information and lacks dedicated uncertainty modeling, limiting its performance in complex dynamic scenes. Furthermore, while DMVFN's reliance on voxel flow for motion estimation may handle large-scale motion well, it may lack accuracy in subtle and complex local motion details. DMVFN trained on Cityscapes exhibits weak generalization ability and often displays shape distortion on untrained datasets.

[0005] Therefore, there is an urgent need to develop entirely new video prediction methods to solve the above problems. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a video prediction method based on a dynamic diffusion model.

[0007] To solve the technical problem, the solution of the present invention is:

[0008] A video prediction method based on a dynamic diffusion model is provided, including the following steps:

[0009] S1, The continuous historical video frame sequence The input routing module outputs a dynamic routing decision vector to control the activation state of each feature extraction module in the feature extraction network.

[0010] S2, A multi-layer feature extraction network is formed by stacking multiple cascaded feature extraction modules, and a multi-scale feature pyramid structure is formed based on the difference in scaling factors; each pair of input frames ( The feature is split into parallel spatial feature branches and motion feature branches. After branch separation and scale transformation, the final output is a fused feature that combines spatial details and motion correlation. ;

[0011] S3, Integrating Features With the last frame in the video frame sequence Encoding fusion is performed to input conditional information into the diffusion model; under the guidance of the conditional information, global conditional control is achieved through cross-attention computation; then, starting from standard Gaussian noise, the predicted optical flow field for the next time step is generated through progressive denoising. ;

[0012] S4. Utilizing the predicted optical flow field The forward warping operation calculates the position of the current pixel in the next time step, and the final predicted frame is synthesized. ;

[0013] S5. Repeat steps S1 to S4 to predict the frame. By connecting with historical video frame sequences, continuous prediction tasks can be achieved.

[0014] As a preferred embodiment of the present invention, in step S1, the current frame in the historical video frame sequence is used. And the previous frame As a pair of input frames, the routing module dynamically generates a binary decision vector for each pair of input frames, which is used to accurately indicate the feature extraction modules that need to be activated in the feature extraction network.

[0015] In a preferred embodiment of the present invention, step S1, in which the output routing decision vector is used to control the activation state of each feature extraction module in the feature extraction network, specifically includes:

[0016] Using raw consecutive video frame pairs ( The input to the routing module is first downsampled by a convolutional encoder to encode the basic spatiotemporal pattern and generate a generalized feature tensor. Then, global average pooling is used to further compress the spatial information into a compact feature vector, forming a summary vector that carries the global motion characteristics of the input frame. Finally, a multilayer perceptron in the form of a fully connected network outputs the final routing decision vector. It is used to directly control the activation state of all feature extraction modules.

[0017] As a preferred embodiment of the present invention, step S2 specifically includes the following steps S21-S24:

[0018] S21. The input is split into two parallel processing flows: a spatial feature branch and a motion feature branch; among them,

[0019] The spatial feature branch is responsible for extracting the global spatial features of the input frame: first, it downsamples through a downsampling convolutional layer to compress the size and extract high-level spatial features; then, it refines the spatial features at the current resolution through a feature extraction convolutional layer.

[0020] The motion feature branch is responsible for extracting motion-related features between frames: first, the downsampling scaling layer downsamples, then the downsampling convolutional layer further downsamples to extract motion features; next, the feature extraction convolutional layer refines the motion features at the current low resolution, and finally, the upsampling scaling layer upsamples to restore the resolution to match the spatial feature branch.

[0021] S22. The features output from the two parallel processing flows in the previous step are input into a concatenation layer to achieve feature fusion; then, a transposed convolutional layer restores a higher resolution through upsampling convolution, outputting features that fuse spatial and motion information. ;

[0022] S23. Among the activated feature extraction modules, the input of the first module is the original consecutive video frame pairs ( The input of the intermediate module is the feature of the output of the previous layer. and video frame pairs ( The output of the final module is a final fused feature that combines spatial detail with motion correlation. ;

[0023] S24. The feature extraction module that is not activated, when receiving features... It is then passed directly to the next layer without any further processing.

[0024] As a preferred embodiment of the present invention, step S3 specifically includes the following steps S31-S34:

[0025] S31. Construct a diffusion model using the U-Net network as the core, and use noisy optical flow data. The corresponding time step codes are used as inputs to the model;

[0026] In the S32 diffusion model, the encoder progressively compresses the input through a series of downsampling layers, extracting noise features at different scales; the decoder progressively restores the noisy latent representation to a clean optical flow prediction through a series of upsampling layers; skip connections directly pass the features from each layer of the encoder to the corresponding layer of the decoder, ensuring that detailed information is preserved during the denoising process; the ultimate goal is to predict the added noise during the forward pass to restore the true optical flow. ;

[0027] S33. The operation of the diffusion model includes a forward diffusion process for adding noise and a generation process for inverse denoising. The former obtains the expected model parameters through training and prepares training data for the latter. The latter starts from standard Gaussian noise and gradually denoises through inference to finally generate the predicted true optical flow. ;

[0028] S34. When reconstructing the optical flow of each local area during the denoising process, conditional information is used to achieve global conditional control through a cross-attention mechanism. This conditional information is the final fused feature output by the dynamic feature extraction network. Compared to the last frame in the original continuous video frames It is formed by the fusion of codes.

[0029] In a preferred embodiment of the present invention, each step of the denoising process follows an iterative formula:

[0030]

[0031] ,

[0032] In the above formulas, Indicates the first The estimated optical flow field of the step; This represents the ratio of the original signal retained in step t; These are predefined noise scheduling parameters used to control the noise intensity at each step; Indicates the first Step-by-step input of noisy optical flow field; Indicates from the first Step to the first The cumulative noise attenuation coefficient of the step; This represents the noise predicted by the denoising network. The standard deviation parameter represents the control of randomness and is used to control the degree of randomness. It uses random noise to ensure the diversity of the generated data.

[0033] As a preferred embodiment of the present invention, the loss of the prediction result is shown in the following formula:

[0034]

[0035] In the formula, This represents the composite loss function used in model training. Indicates future frames With the predicted frame L1 norm distance between them; Represents tag frame With reconstructed frames Structural similarity index between them; Represents a real future video frame; This indicates the predicted frame.

[0036] The present invention also provides a video prediction system for implementing the aforementioned method, the system comprising the following modules arranged in sequence:

[0037] The routing module includes a convolutional encoder, a global average pooling layer, and a multilayer perceptron.

[0038] The feature extraction network is formed by stacking multiple feature extraction modules in series. Each feature extraction module includes two parallel spatial feature branches and a motion feature branch. The former consists of a downsampling scaling layer, a downsampling convolutional layer, a feature extraction convolutional layer, and an upsampling scaling layer, while the latter consists of a downsampling convolutional layer and a feature extraction convolutional layer. The outputs of the two branches are connected in parallel to a concatenation layer and a transposed convolutional layer to achieve feature fusion and result output.

[0039] The diffusion model, whose core architecture is the U-Net network, includes an encoder consisting of a series of downsampling layers and a decoder consisting of a series of upsampling layers. The layers of the encoder and the corresponding layers of the decoder are connected by skip connections to achieve direct feature transfer.

[0040] The frame synthesis module is used to synthesize predicted future frames through forward warping operations.

[0041] In a preferred embodiment of the present invention, the convolutional encoder in the routing module is a lightweight convolutional encoder, consisting of no more than 3 convolutional layers and no more than 2 pooling layers; in the feature extraction network, the number of feature extraction modules is 6-12, and they are sequentially divided into shallow modules, medium modules, and deep modules in a cascaded order, with the ratio factor of the three being... The values ​​are decreasing; scaling factor This determines the scale of the motion feature branch in the feature extraction module.

[0042] Compared with the prior art, the technical advantages of the present invention are:

[0043] 1. The video frame prediction system proposed in this invention has better performance. It can adaptively learn the features of the current and past driving environment using a dynamic network, and use the randomness of the diffusion model to generate predicted optical flow, thereby achieving the purpose of video frame prediction.

[0044] 2. This invention cleverly separates the scene content and motion information in the video, transforming the complex video frame generation problem into an optical flow prediction problem that is easier to model; thus, while ensuring the generation quality, it significantly reduces the computational complexity of the model.

[0045] 3. This invention uses a diffusion model to predict optical flow changes, which can avoid the situation where deterministic prediction methods cannot accurately predict random driving environments, reduce the training and inference costs of the diffusion model, and overcome the drawback of the diffusion model losing information when using latent space to analyze images.

[0046] 4. The prediction system of this invention uses a lightweight dynamic network, which can minimize the number of parameters that the network needs to update while learning features at different scales of the video.

[0047] 5. This invention enhances the randomness of the model by introducing a diffusion model generation mechanism to improve the predictive ability of complex and ever-changing driving environments; and improves the model's adaptability and generalization ability to diverse worlds by training on the OpenDV-2K dataset, which has the widest diversity.

[0048] 6. This invention can generate clearer, more accurate, and more physically consistent future frame prediction results. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of the overall structure of the video frame prediction system in this invention.

[0050] Figure 2 This is a schematic diagram of the Feature Extraction Module (MSFB). Detailed Implementation

[0051] This invention studies the problem of video frame prediction, a branch of computer vision, with the aim of predicting future frames based on existing frame sequences. This invention primarily involves self-supervised deep learning techniques, which analyze and process traffic scene videos from the driver's perspective to achieve early perception of changes in the street environment, thereby improving the safety and reliability of autonomous driving technologies and driver assistance systems.

[0052] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0053] Part One: Implementation Scheme of the Invention

[0054] The video prediction system provided by this invention is as follows: Figure 1 As shown, it includes a routing module, a feature extraction network, a diffusion model, and a frame synthesis module arranged sequentially. Figure 1 In the diagram, the data input to the diffusion model is represented by horizontal bars of three different colors: green, blue, and orange, each corresponding to the current frame image. I t Previous frame image I t-1 and extracted features e final The diffusion model is represented by multiple vertical bars arranged in sequence, illustrating that the U-Net network, which serves as its core architecture, has a layered structure. In the frame synthesis module, the image obtained after denoising by the diffusion model is received, and the predicted optical flow is finally generated.

[0055] 1. RouteModule (routing module)

[0056] In complex autonomous driving scenarios, the motion patterns between adjacent frames vary greatly. For example, a pair of frames depicting vehicles traveling at a constant speed on a highway has relatively simple motion characteristics; while a pair of frames depicting multiple vehicles merging and pedestrians crossing at an urban intersection contains a large amount of complex, multi-scale motion information. If existing technologies use static networks with fixed depth and structure to handle all cases, it will inevitably lead to huge computational redundancy. To address this, this invention introduces an intelligent routing module. The core idea is to allow the network to adaptively determine the computation path based on the complexity of the input content, dynamically activating feature extraction modules of different depths, thereby achieving an optimal balance between accuracy and efficiency.

[0057] (1) Design motivation and objectives

[0058] Using a fixed-depth feature extraction network model has two significant drawbacks: for simple inputs, all layers of the network are activated, resulting in a large amount of unnecessary computation and slow inference speed; the fixed network structure cannot adapt to scenarios of varying complexity, potentially leading to "overfitting" in simple scenarios or "underfitting" in complex scenarios.

[0059] This routing module aims to solve the above problems, with the design goal of providing routing for each pair of input frames (current frame). And the previous frame A binary decision vector is dynamically generated, which precisely indicates which MSFB modules in the network need to be activated for computation and which can be skipped, thereby improving efficiency while maintaining model performance.

[0060] (2) Routing decision generation

[0061] The routing module itself is an extremely lightweight neural network, mainly consisting of a convolutional encoder, a global average pooling layer (GAP), and a multilayer perceptron (MLP).

[0062] Input: The input to the routing module is raw, unprocessed consecutive video frame pairs. This ensures that decisions are based on the most complete, unaltered information about the scenario.

[0063] Lightweight Convolutional Encoding: Video frame pairs first pass through a lightweight convolutional encoder. This encoder consists of no more than 3 convolutional layers and no more than 2 pooling layers. Its purpose is not to extract fine features, but to quickly downsample the input and encode its basic spatiotemporal patterns, generating a low-dimensional, generalized feature tensor.

[0064] Global perception and vector generation: Next, spatial information is compressed into a compact feature vector through a global average pooling operation. This vector carries a summary of the global motion characteristics of the input frame pair.

[0065] Decision generation: This summary vector is fed into a small multilayer perceptron (i.e., a fully connected network). The final layer of the MLP outputs the final routing decision vector directly through a gating function. Where n represents the number of MSFB modules, The activation instruction corresponding to the i-th MSFB module.

[0066] (3) Dynamic execution

[0067] Generated route vectors The activation state of all MSFB modules is directly controlled, therefore the characteristic output of each MSFB module can be represented as:

[0068]

[0069] In the formula, Indicates the first The output characteristics of each MSFB module depend on the routing vector components. The active state; Indicates the first Characteristic transformation functions of each MSFB module; Indicates input to the first Features of each MSFB module; Represents the routing vector The Each component has a value of 0 or 1.

[0070] For scenarios with simple motion (such as cruising at a constant speed in a straight line), the routing vector will contain more 0s, guiding the network to skip a large number of MSFB modules and achieve rapid inference. For scenarios with complex motion (such as intersections), the routing vector will contain more 1s, ensuring that all necessary MSFB modules are activated for deep feature extraction and guaranteeing prediction accuracy.

[0071] A lightweight routing module dynamically activates which depths of MSFB based on the input content, avoiding redundant computation and efficiently extracting fusion features containing rich spatiotemporal information. .

[0072] 2. Feature Extraction Module (Multi-Scale Feature Extraction Block, MSFB)

[0073] One of the core innovations of this invention lies in the design of a highly efficient dynamic feature extraction network that takes into account both motion and spatial features. This network consists of 8-12 (e.g., 10 in the example) MSFB modules arranged in series with the same structure. The core design concept of the MSFB module is to extract the appearance features of the background space and the motion features of objects within the image in parallel through a dual-branch architecture, constructing a multi-scale feature pyramid, thereby achieving comprehensive perception of all information from macroscopic motion to microscopic details.

[0074] (1) Design motivation and input

[0075] Autonomous driving scenarios are complex and varied, encompassing both fast-moving objects like vehicles and pedestrians (large-scale motion) and subtle visual changes such as traffic sign textures, road conditions, and rain / snow (small-scale details). An excellent prediction system must be able to capture both types of information simultaneously. Traditional single-scale convolutional networks struggle to balance receptive field and resolution. Therefore, this invention employs a dual-branch heterogeneous architecture for each MSFB module.

[0076] Each MSFB module receives three inputs: the current frame. Previous frame and the output features of the previous layer For the first MSFB module in the network, its input characteristics are... Initialize to two tensors.

[0077] (2) Multi-scale feature pyramid

[0078] The entire feature extraction network consists of multiple (e.g., 10) stacked MSFB modules. These are sequentially divided into shallow, medium, and deep modules, with the proportions of each layer determined by a factor. The values ​​are in a decreasing order. The scaling factor for each module. These differences allow the network to form a powerful multi-scale feature pyramid:

[0079] Shallow modules ( Larger values ​​(e.g., in the range of 6-10) provide a very large receptive field, focusing on capturing global motion trends such as the movement of distant vehicles and the overall curvature of roads.

[0080] Middle-layer module ( Medium (e.g., values ​​in the range of 2-6), capturing the movement of objects at medium distances, such as nearby vehicles changing lanes or pedestrians on the sidewalk.

[0081] Deep module ( Smaller values ​​(e.g., values ​​in the range of 0-2) focus on repairing and preserving local features, ensuring the accuracy of predictions for small regions.

[0082] This coarse-to-fine pyramid structure enables the model to accurately extract features from both short-term and long-term scene dynamics, providing extremely rich and structured conditional information for subsequent diffusion models, which is a key guarantee for generating high-quality prediction frames.

[0083] (3) Dual-branch processing flow

[0084] Each MSFB module has two branches, each with its own specific function, working together:

[0085] The purpose of the motion feature branch is to capture large-scale motion cues. By downsampling, the receptive field is expanded, making this branch more sensitive to rapid, large-scale displacements of objects in the scene. First, the frames are... and The data is concatenated along the channel dimension to form a tensor containing spatiotemporal information. This tensor is then subjected to a step size of... Downsampling is performed on the convolutional layer (the i-th MSFB). Scale factor The scale of this branch is determined, and its value sequence is predefined as [8, 8, 4, 4, 4, 2, 2, 2, 1, 1].

[0086] The spatial feature branch is specifically designed to preserve high-resolution spatial details and static representation information. It ensures that the generated predicted frames maintain high fidelity in texture, edge, and background details. This branch preserves the original high resolution of the input without any downsampling and consists of two lightweight convolutional layers that focus on extracting shallow but fine-grained features.

[0087] (4) Feature fusion and output

[0088] Although the features from the two branches have different focuses, they describe different aspects of the same scene, thus requiring deep fusion. By concatenating the low-resolution, large receptive field feature map extracted from the motion branch with the high-resolution, detailed feature map extracted from the spatial branch along the channel dimension, the macroscopic motion context and microscopic spatial details are integrated into a unified feature representation. However, since the feature map from the motion branch is relatively small, it needs to be upsampled using a transposed convolution to restore its resolution to match the input before concatenating with the feature map from the spatial branch. The fused features are finally refined and dimensionality-reduced through a fusion convolutional layer to form the output of the current MSFB module. .

[0089] The above process can be formally represented as: ;

[0090] In the formula, Indicates the first The dynamic feature representation output by multiple multi-scale feature modules (MSFB); Indicates the first The feature transformation functions executed by each MSFB module; Indicates input to the first Initial feature representation of each MSFB module.

[0091] (5) Figure 2 Further explanation of the specific example shown

[0092] Figure 2 The MSFB module shown in the image is a neural network module structure that integrates spatial and motion features. Its core is to separate the extraction process of "spatial features" and "motion features" and then fuse them for output. Figure 2 The C inside the middle circle refers to Concatenate (feature splicing / fusion); s in Conv,s=2 and Conv,s=1 represents stride (convolution stride); Si in "1 / Si×scaling" and "2Si×scaling" represents the scaling factor of the sampling.

[0093] The activated module splits the input into two parallel branches: Spatial (spatial feature branch) and Motion (motion feature branch). Among them,

[0094] The Spatial (spatial feature branch) process is responsible for extracting the global spatial features of the input: first, it downsamples (halves the resolution) through a downsampling convolutional layer (convolution with stride=2) to quickly compress the size and extract high-level spatial features; then, it refines the spatial features at the current resolution through a feature extraction convolutional layer (convolution with stride=1).

[0095] The Motion (motion feature branch) process is responsible for extracting motion-related features between frames (while also considering computational efficiency): First, the downsampling scaling layer downsamples by a factor of Si (reducing the size and subsequent computation); the downsampling convolutional layer further downsamples and extracts motion features; the feature extraction convolutional layer refines the motion features at the current low resolution; finally, the upsampling scaling layer upsamples by a factor of 2Si (restoring the resolution to match the Spatial branch).

[0096] Feature fusion and output: Features from the two branches are fused using a concatenation layer; then, a transposed convolutional layer (TransposedConv, i.e., upsampling convolution) restores higher resolution; the final output is e. i+1 Features that combine spatial and motion information (used for subsequent tasks, such as generating the next frame).

[0097] The MSFB module employs a "branch separation processing + scale transformation" approach, simultaneously considering: the spatial branch efficiently extracts global spatial features from a single frame; the motion branch captures inter-frame motion information by first downsampling and subtracting calculations, then upsampling to match dimensions; and finally, it fuses the two types of features to output features that combine spatial details and motion correlation.

[0098] 3. Dynamic diffusion model based on conditional information

[0099] In the fields of computer vision and video processing, optical flow refers to the instantaneous velocity vector field of pixel motion formed on a two-dimensional image plane by the motion of objects or cameras in a three-dimensional scene. Each pixel corresponds to a two-dimensional vector (u, v), representing the displacement and direction of the pixel's motion in the horizontal and vertical directions, respectively. Its core is to quantify the motion information between frames by the pixel changes of consecutive frames.

[0100] The essence of video is a temporally continuous sequence of image frames. The correlation between frames (especially motion correlation) is the core feature that distinguishes video from static images, and optical flow is the key tool for quantifying this motion correlation. The relationship between the two can be understood from three dimensions:

[0101] (1) Optical flow is the "quantization carrier" of video motion information.

[0102] Translation, rotation, and scaling of objects in a video, or camera movements such as panning, tilting, and rotating, all manifest as changes in pixel position between frames. Optical flow uses vector fields to intuitively convert this visual motion into calculable numerical information, allowing machines to "understand" the dynamic patterns in the video. For example, in a video of a car traveling from left to right, the optical flow vectors corresponding to the pixels it covers will all show a consistent horizontal direction to the right, and the vector amplitude corresponds to the car's speed.

[0103] (2) Optical flow is the core "prior basis" of video prediction technology.

[0104] The goal of video prediction is to generate future video frames based on existing historical frames. The core challenge is accurately modeling the motion logic between frames, and optical flow provides direct motion priors.

[0105] In traditional video prediction methods, the optical flow of historical frames is first calculated, and then the optical flow field of future frames is extrapolated based on the motion trend of the optical flow (such as uniform speed or acceleration). Finally, the predicted frame is generated by combining the pixel information of the original frame.

[0106] Video prediction systems in the era of deep learning (such as PredRNN and VideoGPT) will use optical flow as an auxiliary feature input to the model, allowing the model to learn both content features (object appearance) and motion features (optical flow) at the same time, thereby reducing the blurring and misalignment of predicted frames and improving the spatiotemporal consistency of prediction results.

[0107] (3) Optical flow is a "key bridge" for video content understanding.

[0108] Optical flow not only serves video prediction, but also helps models distinguish moving objects in the foreground from static background regions in a video, thereby enabling deeper video content analysis:

[0109] For example, in video prediction tasks for pedestrian detection, optical flow can quickly locate the movement area of ​​pedestrians, allowing the model to prioritize the changes in pedestrian posture and thus more accurately predict their next action.

[0110] For video prediction in dynamic scenes (such as traffic flow and pedestrian flow), the spatial consistency of optical flow can constrain the rationality of motion in the predicted frames and avoid logical errors such as "object teleportation" and "incorrect motion direction".

[0111] Based on this novel understanding and perception of the relationship between video and optical flow, this invention proposes a lightweight diffusion model. After effectively extracting the dependencies and features of past frames, it learns and generates optical flow instead of scene generation. To this end, this invention abandons the deterministic paradigm of directly predicting pixel values ​​in future frames and innovatively applies the diffusion model to optical flow site prediction. The core of this module is learning the denoising process from noise to deterministic optical flow, incorporating conditional information extracted by a dynamic feature network at each step of the denoising process, thereby generating high-quality future optical flow that conforms to physical laws while also incorporating randomness.

[0112] (1) This invention selects the U-Net network as the core architecture of the diffusion model, and adaptively modifies it to form a denoising network for processing image-type data (optical flow fields can be regarded as a special type of image). In this diffusion model (denoising network), the encoder-decoder structure combined with skip connections can effectively capture multi-scale features and preserve detailed information during the denoising process. The specific contents mainly include:

[0113] Input and output: The network at each time step The input is noisy optical flow data. And the corresponding time-step encoding. The goal is to predict the amount added to the actual optical flow during the forward pass. The noise in the image is then restored. .

[0114] Time step integration: Time step The denoising is transformed into a high-dimensional vector through sinusoidal position encoding, which is then injected into each layer of U-Net, enabling the network to perceive the current denoising progress and adjust its denoising behavior accordingly.

[0115] Conditional control mechanism: Conditional information c is derived from historical features extracted by a dynamic feature extraction network and the last video frame. The integration of encoding is crucial. This invention employs a cross-attention mechanism to achieve precise conditional control. Specifically, the conditional information 'c' is used as both a key and a value, and cross-attention is calculated with the intermediate features (as a query) in the U-Net decoder. This allows the denoising network to "pay attention" to the most relevant historical motion and contextual parts of the conditional information when reconstructing the optical flow of each local area, achieving global conditional control.

[0116] Multi-layered feature processing: The U-Net network's encoder progressively compresses the input through a series of downsampling layers, extracting noise features at different scales. The decoder, on the other hand, progressively restores the noisy latent representation to a clean optical flow prediction through a series of upsampling layers. Skip connections directly pass features from each layer of the encoder to the corresponding layer of the decoder, ensuring that detailed information is preserved during denoising.

[0117] (2) Forward diffusion process (noise addition)

[0118] During the training phase of the diffusion model, clean ground truth values ​​of future optical flow are obtained using the traditional algorithm FlowNet, denoted as... The forward diffusion process is a fixed Markov chain that, through... Gaussian noise is gradually added within each step. Destruction is pure noise The process is defined as follows:

[0119]

[0120]

[0121] in, The conditional probability distribution representing the forward diffusion process; This represents a Gaussian distribution (normal distribution). Indicates the first Noisy optical flow during the step; Represents the actual future optical flow (truth value); This represents the cumulative noise attenuation coefficient (or noise dissipation product). This indicates that the covariance matrix is ​​an identity matrix, meaning that the added noise is independent and homoscedastic in each dimension; Indicates the first Signal retention rate at each step; , is the diffusion time step; This represents the scheduling of a predefined noise variance, controlling the amount of noise added at each step. This process requires no learning; its purpose is to prepare training data pairs (noise data) for the reverse denoising process. and clean optical flow target ).

[0122] In the reverse denoising training process, minimizing the mean square error between the predicted noise and the actual noise is taken as the training objective. By optimizing this objective, the network learns how to accurately remove noise, thereby gradually recovering clean optical flow.

[0123] The loss from the prediction result can be expressed as L1 loss (preserving edges) and SSIM loss (preserving structure), as follows:

[0124]

[0125] In the formula, It is the total loss function, representing the composite loss function for model training; It is the L1 loss (mean absolute error) term, representing the future frame. With the predicted frame L1 norm distance between them (pixel-level absolute error); Represents the SSIM (structural similarity) loss term; SSIM (Structural Similarity) represents the tag frame. With reconstructed frames Structural similarity index between them; Indicates the model prediction Momentary video frames; express Realistic future video frames (truth values).

[0126] By directly calculating the sum of the absolute differences of each pixel value between the predicted frame and the real frame, accuracy in color and brightness is ensured, and sharp edges are tended to be preserved. SSIM loss corresponds to perceptual-level structural similarity constraints. By simulating the human visual system, it compares the similarity of two images in terms of brightness, contrast, and structure, promoting visually more natural and texture-clearer generated results while effectively reducing blur.

[0127] The gradients of all trainable parameters (including MSFB weights, routing module weights, and diffusion model weights) are calculated, and the Adam gradient descent algorithm is used to update the parameters through backpropagation. Training continues until the loss converges, ultimately resulting in a video prediction system with high generation quality and computational efficiency. This part is prior art and will not be elaborated upon in this invention.

[0128] (3) Generation process (reverse denoising)

[0129] The inverse process is crucial for achieving high-quality predictions. During inference, the diffusion model needs to start with standard Gaussian noise, progressively denoise it, and finally generate the predicted optical flow. Unlike unconditional diffusion, the denoising process of this invention is performed at each time step. All are guided by conditional information c. Conditional information c consists of historical features extracted by the dynamic feature extraction network and the last video frame. It is encoded and merged, encapsulating the motion history and spatial context of past frames.

[0130] The denoising process is implemented using a parameterized denoising network, U-Net, with the goal of predicting the noise added to the data. The network uses time steps... The sinusoidal positional encoding is transformed into a high-dimensional vector and fused with features. The summation serves as input, enabling the network to perceive the current denoising progress. The conditional information 'c' acts as both a key and a value, and undergoes cross-attention computation with the features in the U-Net decoder (as a query). This allows each position during the denoising process to "focus" on the most relevant part of the conditional information, achieving precise conditional control.

[0131] Each step of the denoising process follows an iterative formula:

[0132]

[0133] ,

[0134] In the above formulas, Indicates the first The estimated optical flow field of the step; This represents the ratio of the original signal retained in step t; These are predefined noise scheduling parameters used to control the noise intensity at each step; Indicates the first Step-by-step input of noisy optical flow field; Indicates from the first Step to the first The cumulative noise attenuation coefficient of the step; This represents the noise predicted by the denoising network. The standard deviation parameter represents the control of randomness and is used to control the degree of randomness. It uses random noise to ensure the diversity of the generated data.

[0135] By accurately removing noise using a diffusion model and gradually recovering clean optical flow, it is possible to achieve [the desired result] for the current frame. To future frames Optical flow field .

[0136] 4. Frame Composition Module

[0137] The core task of the frame synthesis module is to utilize the optical flow field. , the known current frame Transformed into predicted future frames This process is achieved through a forward warping operation, which directly calculates the position of the current pixel at the next time step using the optical flow field, thereby obtaining the prediction result. In other words, it utilizes the generated high-quality optical flow. By performing a forward warp operation, the current frame The pixels in the image are "moved" to the predicted positions guided by optical flow, thus synthesizing the final predicted frame. ,Right now .

[0138] Specifically, the frame synthesis module receives the current frame image. and model predictions from Time's up Light flow field at any moment Using these two inputs, a forward warp operation is performed to synthesize the predicted future frame. Specifically, it includes the following steps:

[0139] (1) Coordinate mapping: based on the optical flow field Motion vector of each pixel Calculate the current frame medium pixel Target location in the next frame .

[0140] (2) Sampling and assignment: exist Pixel values ​​assigned exist The pixel value.

[0141] (3) Hole handling: Compared with discretization and occlusion, there will be unmapped pixels (holes) in the target image. This module uses a bilinear interpolation background filling strategy to solve this problem and ensure the continuity of the generated image.

[0142] 5. Video Prediction Methods

[0143] Based on the detailed descriptions of the routing module, feature extraction network, diffusion model, and frame synthesis module in the video prediction system above, the specific operation steps of the video prediction method based on the dynamic diffusion model described in this invention are summarized as follows:

[0144] S1, The continuous historical video frame sequence The input routing module outputs a dynamic routing decision vector to control the activation state of each feature extraction module in the feature extraction network.

[0145] The current frame in the historical video frame sequence And the previous frame As a pair of input frames, the routing module dynamically generates a binary decision vector for each pair of input frames, which is used to accurately indicate the feature extraction modules that need to be activated in the feature extraction network.

[0146] Using raw consecutive video frame pairs ( The input to the routing module is first downsampled by a convolutional encoder to encode the basic spatiotemporal pattern and generate a generalized feature tensor. Then, global average pooling is used to further compress the spatial information into a compact feature vector, forming a summary vector that carries the global motion characteristics of the input frame. Finally, a multilayer perceptron in the form of a fully connected network outputs the final routing decision vector. It is used to directly control the activation state of all feature extraction modules.

[0147] S2, A multi-layer feature extraction network is formed by stacking multiple cascaded feature extraction modules, and a multi-scale feature pyramid structure is formed based on the difference in scaling factors; each pair of input frames ( The feature is split into parallel spatial feature branches and motion feature branches. After branch separation and scale transformation, the final output is a fused feature that combines spatial details and motion correlation. ;

[0148] Step S2 specifically includes steps S21-S24 as shown below:

[0149] S21. The input is split into two parallel processing flows: spatial feature branch and motion feature branch.

[0150] The spatial feature branch is responsible for extracting the global spatial features of the input frame: first, it downsamples through a downsampling convolutional layer to compress the size and extract high-level spatial features; then, it refines the spatial features at the current resolution through a feature extraction convolutional layer. The motion feature branch is responsible for extracting motion-related features between frames: first, it downsamples through a downsampling scaling layer, then further downsamples through a downsampling convolutional layer to extract motion features; then, it refines the motion features at the current low resolution through a feature extraction convolutional layer, and finally, it upsamples through an upsampling scaling layer to restore the resolution to match that of the spatial feature branch.

[0151] S22. The features output from the two parallel processing flows in the previous step are input into a concatenation layer to achieve feature fusion; then, a transposed convolutional layer restores a higher resolution through upsampling convolution, outputting features that fuse spatial and motion information. ;

[0152] S23. Among the activated feature extraction modules, the input of the first module is the original consecutive video frame pairs ( The input of the intermediate module is the feature of the output of the previous layer. and video frame pairs ( The output of the final module is a final fused feature that combines spatial detail with motion correlation. ;

[0153] S24. The feature extraction module that is not activated, when receiving features... It is then passed directly to the next layer without any further processing.

[0154] S3, Integrating Features With the last frame in the video frame sequence Encoding fusion is performed to input conditional information into the diffusion model; under the guidance of the conditional information, global conditional control is achieved through cross-attention computation; then, starting from standard Gaussian noise, the predicted optical flow field for the next time step is generated through progressive denoising. ;

[0155] Step S3 specifically includes steps S31-S34 as shown below:

[0156] S31. Construct a diffusion model using the U-Net network as the core, and use noisy optical flow data. The corresponding time step codes are used as inputs to the model;

[0157] In the S32 diffusion model, the encoder progressively compresses the input through a series of downsampling layers, extracting noise features at different scales; the decoder progressively restores the noisy latent representation to a clean optical flow prediction through a series of upsampling layers; skip connections directly pass the features from each layer of the encoder to the corresponding layer of the decoder, ensuring that detailed information is preserved during the denoising process; the ultimate goal is to predict the added noise during the forward pass to restore the true optical flow. ;

[0158] S33. The operation of the diffusion model includes a forward diffusion process for adding noise and a generation process for inverse denoising. The former obtains the expected model parameters through training and prepares training data for the latter. The latter starts from standard Gaussian noise and gradually denoises through inference to finally generate the predicted true optical flow. ;

[0159] S34. When reconstructing the optical flow of each local area during the denoising process, conditional information is used to achieve global conditional control through a cross-attention mechanism. This conditional information is the final fused feature output by the dynamic feature extraction network. Compared to the last frame in the original continuous video frames It is formed by the fusion of codes.

[0160] S4. Utilizing the predicted optical flow field The forward warping operation calculates the position of the current pixel in the next time step, and the final predicted frame is synthesized. ;

[0161] S5. Repeat steps S1 to S4 to predict the frame. By connecting with historical video frame sequences, continuous prediction tasks can be achieved.

[0162] Extensive experiments on standard autonomous driving datasets such as KITTI and Cityscapes demonstrate that this invention has achieved state-of-the-art performance in key evaluation metrics, as shown in Table 1.

[0163] Table 1 Training Results of Each Model

[0164]

[0165] Two datasets were used in the testing experiments: Cityscapes is a large-scale pixel-level labeled dataset for urban street scene understanding, containing images and videos of real driving scenarios, and is a commonly used benchmark in fields such as autonomous driving and image generation. KITTI (Karlsruhe Institute of Technology and Toyota Technological Institute) is the most classic multimodal benchmark dataset in the field of autonomous driving, containing data such as images, LiDAR, and 3D annotations, and is used to test tasks such as perception and localization.

[0166] The training methods include: DVF (Deep Voxel Flow), MCVD (Masked Conditional Video Diffusion), DMVFN (Dynamic Multi-scale Voxel Flow Network), and STDiff (Spatio-temporal Diffusio, a video generation method based on a diffusion model).

[0167] The evaluation metrics included: FVD (Fréchet Video Distance), where a smaller value (↓) indicates that the generated video is closer to the real video and of better quality; SSIM (Structural Similarity Index), where a larger value (↑) indicates that the generated image is more consistent with the structure of the real image and has a better visual effect; and LPIPS (Learned Perceptual Image Patch Similarity), where a smaller value (↓) indicates that the generated image is closer to the real image at the human perception level.

[0168] As shown in Table 1, the proposed method achieves a 15%-25% reduction in error compared to traditional deterministic prediction systems (such as DVF) on the Fréchet VideoDistance (FVD) metric, which measures the overall realism and temporal consistency of videos. On the perceptual similarity metric LPIPS, it also achieves a 5%-10% improvement compared to other diffusion-based video prediction systems (such as MCVD). This indicates that the generated frames are closer to the real world in terms of texture, edge details, and other aspects, resulting in a better visual experience for the human eye. Through multi-scale feature pyramids and spatial feature branches, the model demonstrates a strong ability to preserve static scene structure and subtle dynamic details (such as traffic sign textures, road surface reflections, and pedestrian outlines). Its Structural Similarity Index (SSIM) consistently remains at a high level (e.g., >0.85 on the Cityscapes dataset), effectively avoiding the blurring and distortion problems common in traditional methods.

[0169] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A video prediction method based on a dynamic diffusion model, characterized in that, Includes the following steps: S1, The continuous historical video frame sequence The input routing module outputs a dynamic routing decision vector to control the activation state of each feature extraction module in the feature extraction network. S2, A multi-layer feature extraction network is formed by stacking multiple cascaded feature extraction modules, and a multi-scale feature pyramid structure is formed based on the difference in scaling factors; each pair of input frames ( The feature is split into parallel spatial feature branches and motion feature branches. After branch separation and scale transformation, the final output is a fused feature that combines spatial details and motion correlation. ; S3, Integrating Features With the last frame in the video frame sequence Encoding fusion is performed to input conditional information into the diffusion model; under the guidance of the conditional information, global conditional control is achieved through cross-attention computation; then, starting from standard Gaussian noise, the predicted optical flow field for the next time step is generated through progressive denoising. Specifically, it includes: S31. Construct a diffusion model using the U-Net network as the core, and use noisy optical flow data. The corresponding diffusion time step encoding is used as the input to the model; In the S32 diffusion model, the encoder compresses the input step by step through a series of downsampling layers to extract noise features at different scales; the decoder gradually restores the noisy latent representation to a clean optical flow prediction through a series of upsampling layers. Skip connections directly pass features from each layer of the encoder to the corresponding layer of the decoder, ensuring that detailed information is preserved during denoising; the ultimate goal is to predict the added noise during the forward pass to restore the true optical flow. ; S33. The operation of the diffusion model includes a forward diffusion process for adding noise and a generation process for inverse denoising. The former obtains the expected model parameters through training and prepares training data for the latter. The latter starts from standard Gaussian noise and gradually denoises through inference to finally generate the predicted true optical flow. ; S34. When reconstructing the optical flow of each local area during the denoising process, conditional information is used to achieve global conditional control through a cross-attention mechanism. This conditional information is the final fused feature output by the dynamic feature extraction network. Compared to the last frame in the original continuous video frames It is formed by the fusion of codes; Each step of the denoising process follows an iterative formula: ； ， ； In the above formulas, Indicates the first Step-by-step diffusion-denoised optical flow field estimation; Indicates the first The ratio of the original signal retained in the step; It is a predefined first Step noise scheduling parameters are used to control the noise intensity of each step; Indicates the first Step-by-step input of noisy optical flow field; Indicates from the first Step to the first The cumulative noise attenuation coefficient of the step; This represents the noise predicted by the denoising network. The standard deviation parameter represents the control of randomness and is used to control the degree of randomness. It uses random noise to ensure the diversity of the generated data; S4. Utilizing the predicted optical flow field The forward warping operation calculates the position of the current pixel in the next time step, and the final predicted frame is synthesized. ; S5. Repeat steps S1 to S4 to predict the frame. By connecting with historical video frame sequences, continuous prediction tasks can be achieved.

2. The method according to claim 1, characterized in that, In step S1, the current frame from the historical video frame sequence is used. And the previous frame As a pair of input frames, the routing module dynamically generates a binary decision vector for each pair of input frames, which is used to accurately indicate the feature extraction modules that need to be activated in the feature extraction network.

3. The method according to claim 1, characterized in that, In step S1, the output routing decision vector controls the activation state of each feature extraction module in the feature extraction network, specifically including: Using raw consecutive video frame pairs ( The input to the routing module is first downsampled by a convolutional encoder to encode the basic spatiotemporal pattern and generate a generalized feature tensor. Then, global average pooling is used to further compress the spatial information into a compact feature vector, forming a summary vector that carries the global motion characteristics of the input frame. Finally, a multilayer perceptron in the form of a fully connected network outputs the final routing decision vector. It is used to directly control the activation state of all feature extraction modules.

4. The method according to claim 1, characterized in that, Step S2 specifically includes the following steps S21-S24: S21. The input is split into two parallel processing flows: spatial feature branch and motion feature branch. in, The spatial feature branch is responsible for extracting the spatial global features of the input frame: first, it uses a downsampling convolutional layer to downsample, compress the size, and extract high-level spatial features; Then, through a feature extraction convolutional layer, the spatial features are refined at the current resolution; The motion feature branch is responsible for extracting motion-related features between frames: first, the downsampling scaling layer downsamples, then the downsampling convolutional layer further downsamples to extract motion features; next, the feature extraction convolutional layer refines the motion features at the current low resolution, and finally, the upsampling scaling layer upsamples to restore the resolution to match the spatial feature branch. S22. The features output from the two parallel processing flows in the previous step are input into a concatenation layer to achieve feature fusion; then, a transposed convolutional layer restores a higher resolution through upsampling convolution, outputting features that fuse spatial and motion information. ; S23. Among the activated feature extraction modules, the input of the first module is the original consecutive video frame pairs ( The input of the intermediate module is the feature of the output of the previous layer. and video frame pairs ( The output of the final module is a final fused feature that combines spatial detail with motion correlation. ; S24. The feature extraction module that is not activated, when receiving features... It is then passed directly to the next layer without any further processing.

5. The method according to claim 1, characterized in that, In step S3, the loss for generating the final prediction result is shown in the following formula: ； In the formula, This represents the composite loss function used in model training. Indicates future frames With the predicted frame L1 norm distance between them; Represents tag frame With reconstructed frames Structural similarity index between them; Represents a real future video frame; This indicates the predicted frame.

6. A video prediction system for implementing the method of any one of claims 1 to 5, characterized in that, The system comprises the following modules arranged in sequence: The routing module includes a convolutional encoder, a global average pooling layer, and a multilayer perceptron. The feature extraction network is formed by stacking multiple feature extraction modules in series. Each feature extraction module includes two parallel spatial feature branches and a motion feature branch. The former consists of a downsampling convolutional layer and a feature extraction convolutional layer, while the latter consists of a downsampling scaling layer, a downsampling convolutional layer, a feature extraction convolutional layer, and an upsampling scaling layer. The outputs of the two branches are connected in parallel to a concatenation layer and a transposed convolutional layer to achieve feature fusion and result output. The diffusion model, whose core architecture is the U-Net network, includes an encoder consisting of a series of downsampling layers and a decoder consisting of a series of upsampling layers. The layers of the encoder and the corresponding layers of the decoder are connected by skip connections to achieve direct feature transfer. The frame synthesis module is used to synthesize predicted future frames through forward warping operations.

7. The video prediction system according to claim 6, characterized in that, The convolutional encoder in the routing module is a lightweight convolutional encoder, consisting of no more than 3 convolutional layers and no more than 2 pooling layers. In the feature extraction network, the number of feature extraction modules is 6-12, and they are sequentially divided into shallow modules, medium modules, and deep modules, with the ratio factor of the three being... The values ​​are in a decreasing order; Scale factor This determines the scale of the motion feature branch in the feature extraction module.

Images