Trajectory control video generation method and device based on depth information and time-frequency optimization

Abstract

The present invention provides a method and device for generating trajectory control videos based on depth information and time-frequency optimization, relating to the field of image processing technology. The present invention optimizes 3D trajectories through multi-entity segmentation, depth estimation, and time-frequency decomposition in combination with user instructions, and uses a multi-scale fusion network to generate control signals. Finally, these signals and the original image are input into an improved Stable Video Diffusion model to generate a video potential representation sequence. This solves the problems of insufficient control accuracy and poor cross-frame consistency of dynamic entity motion in existing video generation methods. Through the 3D trajectory modeling guided by depth information and the time-frequency joint optimization mechanism, the motion smoothness, spatial realism, and time-frequency stability of the generated video are significantly improved.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and video generation, and in particular to a method and device for generating trajectory-controlled videos based on depth information and time-frequency optimization. Background Art

[0002] Existing video generation methods mainly rely on 2D image features or rough motion estimation, making it difficult to accurately control the 3D motion of dynamic entities, and there is a common problem of insufficient cross-frame consistency. Traditional methods (such as motion modeling based on keyframe interpolation) can achieve basic motion control by manually defining trajectory points or relying on sparse optical flow fields, but they cannot effectively capture the depth information and complex motion patterns of multiple entities, resulting in the accumulation of 3D trajectory reconstruction errors, which are specifically manifested as depth jump noise (such as unreasonable Z-axis jitter when the entity moves) and motion paths that deviate from physical laws (such as the lack of continuity in the deformation of non-rigid objects). In recent years, although deep learning-based video generation models (such as diffusion models and GANs) have improved in visual effects, their motion control still has significant limitations: the problem of insufficient accuracy in dynamic entity control mainly stems from the neglect of depth information and the coarseness of control point sampling. For example, existing methods mostly use feature alignment in 2D pixel space (such as the optical flow-guided attention mechanism) without introducing a depth estimation network, which leads to projection conflicts between foreground and background entities in 3D space and motion occlusion artifacts (such as an object passing through the surface of another object). At the same time, the uniform grid sampling strategy has difficulty covering highly motion-sensitive areas such as joints and edges, resulting in the loss of local motion details.

[0003] Regarding cross-frame consistency, existing methods often rely on temporal convolution or recurrent networks to model inter-frame correlations, but their ability to constrain high-frequency motion components (such as rapid jitter) is weak. LSTM-based models are prone to motion amplitude decay (e.g., entities gradually coming to a standstill) in long video generation, while the Transformer architecture, limited by the computational complexity of its self-attention mechanism, struggles to achieve joint optimization in the time and frequency domains. Furthermore, the frequency-domain energy distribution (e.g., amplitude spectrum) of the generated video exhibits abrupt changes across frames, causing high-frequency flicker noise (e.g., inconsistent lighting or jittery textures). Existing frameworks (e.g., Stable Video Diffusion) directly concatenate control signals (e.g., segmentation masks) to the input, but lack multi-scale feature fusion mechanisms. This results in gradual attenuation of the control signal during latent space diffusion, and the annealing sampling phase is prone to oversmoothing effects (e.g., blurred edges) or motion drift (e.g., entities deviating from their intended path). Recent research has attempted to enhance control accuracy through monocular depth estimation, but has not combined time-frequency analysis to suppress high-frequency noise. Furthermore, K-means clustering has been used to extract control points, but the lack of spatial Gaussian weighting guidance causes control points to stray from key kinematic regions.

[0004] Therefore, existing video generation methods have problems such as insufficient accuracy in controlling dynamic entity motion and poor cross-frame consistency. The above technical defects highlight the necessity of integrating depth information, time-frequency optimization, and multi-scale control, and also provide improvement directions for this application. Summary of the Invention

[0005] In response to the above technical problems in the related art, the present invention proposes a method and device for generating trajectory control videos based on depth information and time-frequency optimization.

[0006] In a first aspect, the present invention provides a method for generating a trajectory control video based on depth information and time-frequency optimization, comprising the following steps:

[0007] S1. Perform multi-entity instance segmentation on the original image I to generate a multi-entity instance mask set M, and dynamically determine the number of control points based on the area of ​​the entity region to extract a multi-scale control point set C covering the key motion area through weighted clustering; the multi-scale control point set C is a 2D control point set;

[0008] S2. Extract the image depth map D through the depth estimation network based on the multi-scale control point set C, and map the multi-scale control point set C combined with the depth value into a global 3D trajectory set T;

[0009] S3, use discrete wavelet transform to perform time-frequency decomposition on the global 3D trajectory set T to obtain low-frequency approximate components and high-frequency detail components, combined with the user direction command U and the adaptive direction gain Adjust the high-frequency detail components to optimize the global 3D trajectory set T to obtain the optimized 3D trajectory set T′;

[0010] S4: Input the multi-entity instance mask set M, the global 3D trajectory set T, and the optimized 3D trajectory set T′ into the multi-scale fusion network to generate the entity-level optical flow field O. i With multi-scale features And the entity-level optical flow field O is fused through the gated cross-scale attention mechanism i With multi-scale features Generate a multi-scale control signal S;

[0011] S5. Input the multi-scale control signal S and the original image I into the improved Stable Video Diffusion model. In the latent space diffusion process of the improved Stable Video Diffusion model, the video potential representation sequence is generated through annealing sampling and cross-modal attention.

[0012] Specifically, step S1 includes the following steps:

[0013] S11, the original image Input to the pre-trained Mask R-CNN instance segmentation network to generate a set of multi-entity instance masks Among them, H is the image height, W is the image width; 3 is the RGB three-channel; entity instance mask element represents the pixel-level segmentation result of the i-th entity in the j-th frame, where i∈[1,N] represents the entity index and N is the total number of entities;

[0014] S12. Mask elements for each entity instance Calculate its effective pixel area And dynamically determine the number of control points based on area ratio And when the rate of change of the entity mask area Forced setting Where α is the empirical parameter of the density of equilibrium control points; Indicates taking The largest element value in Indicates taking The smallest element value in ; represents the binary instance mask matrix of the i-th entity in the j-th frame;

[0015] S13, effective pixel coordinate set for mask Perform weighted K-means clustering and output the cluster center set after iterative convergence Finally, a set of multi-scale control points covering the key motion area is generated Where N is the total number of entities, Represents the 2D control point coordinate set of the i-th entity in the j-th frame; the weight function of the weighted K-means clustering adopts spatial Gaussian distribution Among them, μ is the centroid of the mask, Controlling the spatial attenuation rate; the key motion areas include edges, joints, significant deformation areas and dynamic texture areas.

[0016] Specifically, step S2 includes the following steps:

[0017] S21, set the multi-scale control points Input to the depth estimation network DepthAnythingV2, through its encoder-decoder structure Extract the dense depth map D of the original image I; where N is the total number of entities; Represents the 2D control point coordinate set of the i-th entity in the j-th frame;

[0018] S22, use the bilinear interpolation formula for each control point Assigning Depth Values Then construct the 3D trajectory point of the i-th entity in the j-th frame And through the timing smoothing constraint Eliminate depth jump noise and finally output a global 3D trajectory set in Represents the 3D trajectory matrix of the i-th entity in F frames; F is the total number of video frames.

[0019] Specifically, step S3 includes the following steps:

[0020] S31, the global 3D trajectory collection Timing direction control instructions input by the user The signals are input to the time-frequency analysis module, and the trajectory signal is decomposed into multiple scales by discrete wavelet transform to obtain the low-frequency approximate component. and high-frequency detail components satisfy in represents the 3D trajectory matrix of the i-th entity in the F frame; is the unit direction vector of the i-th entity in the j-th frame; L is the number of decomposition layers;

[0021] S32, based on the timing direction control instruction U, the high frequency detail component Perform directional weighted adjustment, specifically: for the lth layer detail component Apply adaptive directional gain At the same time, through the threshold function Suppress noise and reconstruct optimized trajectory The final output is the optimized 3D trajectory set after time-frequency optimization Among them, ⊙ represents element-by-element multiplication, is the layer-dependent gain coefficient.

[0022] Specifically, step S4 includes the following steps:

[0023] S41. Mask the entity instance collection Global 3D trajectory collection And optimize the 3D trajectory set T′={T′ i} are jointly input into a multi-scale control feature fusion network; the multi-scale control feature fusion network consists of a cascaded encoder-decoder structure and a gated cross-scale attention module;

[0024] S42, based on each entity trajectory in T' The optical flow generation module calculates the frame-by-frame dense optical flow field O i , the frame-by-frame dense optical flow field O i The value of the optimized 3D trajectory point Projection transformation in camera coordinate system Determine; among them is the camera internal parameter, R∈SO(3), is an external parameter; the optimized 3D trajectory point

[0025] S43. Construct a multi-scale feature pyramid: Extract hierarchical features from the original image I through the ResNet-50 backbone network And the optical flow field O i Generate by downsampling to the corresponding scale through bilinear interpolation And in each scale l, the multi-scale features are fused into the hierarchical features through the gated cross-scale attention module

[0026] The formula of the gated cross-scale attention module is as follows:

[0027]

[0028] Where W g,l 、W a,l 、 is the learnable weight matrix, represents the convolution operation, [·] is channel concatenation, and ⊙ is element-by-element multiplication; C l is the number of channels of the l-th layer feature map extracted by the ResNet-50 backbone network, and its value is determined by the network structure; Represents the downsampled optical flow field of the i-th entity at the l-th scale;

[0029] S44, multi-scale features through skip connection It is concatenated with the decoder upsampled features and fused with convolution kernels to output spatially aligned multi-scale control signals. It satisfies the spatial alignment constraint Where D is the control signal dimension; the convolution kernel is a 3×3 convolution kernel; MLP is a multi-layer perceptron; S j (x,y) represents the motion control strength weight of the j-th frame at the spatial coordinate (x,y).

[0030] Specifically, step S5 includes the following steps:

[0031] S51, multi-scale control signal With the original image Input to the improved model based on the Stable Video Diffusion framework, first compressing the input image I into a potential representation through the 3D-VAE encoder At the same time, the multi-scale control signal S is mapped into a conditional embedding through the spatiotemporal convolutional network. In the diffusion denoising process, the latent variable z tThrough the iterative update of the improved UNet architecture, the time-dependent residual block calculation form is:

[0032]

[0033] Among them, AdaGN(z t ,t) is the embedding vector injected at time step t, representing the adaptive group normalization layer; CrossAttn(z t ,c) is the cross-modal attention module, used to calculate z t c interaction weight And weighted fusion as conditional features; h = H / 8; w = W / 8; d = 4; t∈[1,T]; Represents the output of the denoising network; Conv 3D It is a 3D convolutional layer that jointly models motion continuity in the spatial dimension (H×W) and the temporal dimension (implicit inter-frame stacking);

[0034] S52, introduce annealing sampling strategy: in the denoising step t∈[T c ,T end ], and in t∈[1,T c ] gradually attenuates the weight of the multi-scale control signal Sγ(t)=min(1,(T c -t) / (T c -1)) to eliminate artifacts caused by over-constraints; finally, the potential representation is stabilized by the frequency domain stabilization module Post-processing outputs temporally smoothed video latent representation sequence is a learnable frequency domain filter; T end represents the termination time step of the annealing sampling strategy; T c Indicates the starting time step of the annealing sampling strategy.

[0035] Specifically, the method further includes:

[0036] S6. Video potential representation sequence Each frame image is divided into local blocks and decomposed into amplitude spectrum M and phase spectrum P through FFT transform. The amplitude spectrum is adaptively smoothed and optimized based on the cross-frame consistency constraint and combined with the phase spectrum P. The inverse FFT is used to restore the time domain image block. The complete video frame is reconstructed by the weighted overlap method to output the final video sequence X′ final .

[0037] Specifically, step S6 includes the following steps:

[0038] S61, video potential representation sequence Input to the frequency domain stabilization module, the video flicker and motion artifacts are eliminated by time-frequency joint, and then the video potential representation sequence is converted into Each frame image is divided into local blocks and decomposed into amplitude spectrum M and phase spectrum P through FFT transformation;

[0039] S62. Based on the cross-frame frequency domain consistency constraint, the amplitude spectrum is adaptively smoothed: the neighborhood mean is calculated using a sliding window, and the local and global frequency domain features are fused through learnable spectral attention weights to generate an optimized amplitude spectrum M′ that suppresses high-frequency noise. The optimized amplitude spectrum M′ is combined with the phase spectrum P, and the time domain image block is restored through inverse FFT. The complete video frame is reconstructed through the weighted overlap method to output the final video sequence X′. final .

[0040] Specifically, step S11 includes the following steps:

[0041] S111, the original image Input to the pre-trained MaskR-CNN backbone network, and extract multi-scale feature maps through convolutional layers and feature pyramid networks (FPN) Where l represents the feature pyramid level; the backbone network of the pre-trained Mask R-CNN model is ResNet-101-FPN, and the downsampling ratio is 2 l ;

[0042] S112, based on the feature pyramid, the region proposal network generates candidate regions through a sliding window The overlapping areas are filtered by non-maximum suppression, and the top N largest candidate areas are retained; where K is the initial number of candidates, (x n ,y n ) is the center coordinate of the candidate region, (w n ,h n ) is the width and height of the candidate region; the threshold value θ of non-maximum suppression NMS Take 0.7;

[0043] S113, use the RoIAlign layer to map the N largest candidate regions to the multi-scale feature map, and extract the fixed-size regional features F roi Input to the fully connected classification branch and regression branch, and then filter twice by NMS to obtain the final entity candidate frame set.

[0044] S114. For each final entity candidate box set The segmentation head generates a binary mask through a deconvolution layer and pixel-wise Sigmoid activation Then upsample to the original resolution H×W by bilinear interpolation to generate a set of multi-entity instance masks aligned with the input image

[0045] In a second aspect, the present invention provides a trajectory control video generation device based on depth information and time-frequency optimization, based on the trajectory control video generation method based on depth information and time-frequency optimization described in the first aspect, comprising the following units:

[0046] A control point generation unit is configured to perform multi-entity instance segmentation on the original image I to generate a multi-entity instance mask set M, dynamically determine the number of control points based on the area of ​​the entity region, and extract a multi-scale control point set C covering the key motion region through weighted clustering; the multi-scale control point set C is a 2D control point set;

[0047] A global 3D trajectory generation unit is used to extract an image depth map D based on a multi-scale control point set C through a depth estimation network, and map the multi-scale control point set C combined with the depth value into a global 3D trajectory set T;

[0048] 3D trajectory optimization unit, used to use discrete wavelet transform to perform time-frequency decomposition on the global 3D trajectory set T to obtain low-frequency approximate components and high-frequency detail components, combined with the user direction instruction U and the adaptive direction gain Adjust the high-frequency detail components to optimize the global 3D trajectory set T to obtain the optimized 3D trajectory set T′;

[0049] The multi-scale control signal generation unit is used to input the multi-entity instance mask set M, the global 3D trajectory set T and the optimized 3D trajectory set T′ into the multi-scale fusion network to generate the entity-level optical flow field O i With multi-scale features And the entity-level optical flow field O is fused through the gated cross-scale attention mechanism i With multi-scale features Generate a multi-scale control signal S;

[0050] The video potential representation sequence generation unit is used to input the multi-scale control signal S and the original image I into the improved Stable Video Diffusion model. The video potential representation sequence is generated by annealing sampling and cross-modal attention in the latent space diffusion process of the improved Stable Video Diffusion model.

[0051] Video denoising unit, used to transform the video potential representation sequence Each frame image is divided into local blocks and decomposed into amplitude spectrum M and phase spectrum P through FFT transform. The amplitude spectrum is adaptively smoothed and optimized based on the cross-frame consistency constraint and combined with the phase spectrum P. The inverse FFT is used to restore the time domain image block. The complete video frame is reconstructed by the weighted overlap method to output the final video sequence X′ final .

[0052] The present invention optimizes 3D trajectories through multi-entity segmentation, depth estimation, and time-frequency decomposition, combined with user instructions, and uses a multi-scale fusion network to generate control signals. Finally, these signals are input into the improved Stable Video Diffusion model with the original image to generate a video potential representation sequence. This solves the problems of insufficient control accuracy of dynamic entity motion and poor cross-frame consistency in existing video generation methods. Through the depth information-guided 3D trajectory modeling and time-frequency joint optimization mechanism, the motion smoothness, spatial realism, and time-frequency stability of the generated video are significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1 A schematic diagram of a trajectory control video generation method based on depth information and time-frequency optimization provided by an embodiment of the present invention;

[0055] Figure 2 Schematic diagram of a trajectory control video generation device based on depth information and time-frequency optimization provided by an embodiment of the present invention;

[0056] Figure 3 Schematic diagram of a trajectory control video generation device based on depth information and time-frequency optimization provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The present invention can be explained in detail through the following examples. The purpose of providing the present invention is to protect all technical improvements within the scope of the present invention. In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" can explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "multiple" and "plurality" is two or more, unless otherwise clearly and specifically defined.

[0058] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0059] Example 1

[0060] refer to Figure 1This embodiment provides a method for generating a trajectory control video based on depth information and time-frequency optimization, comprising the following steps:

[0061] S1. Perform multi-entity instance segmentation on the original image I to generate a multi-entity instance mask set M, and dynamically determine the number of control points based on the area of ​​the entity region to extract a multi-scale control point set C covering the key motion area through weighted clustering; the multi-scale control point set C is a 2D control point set;

[0062] The original image I is an image corresponding to a video frame in a video sequence; the total number of frames in the video sequence is F (i.e., the length of the time dimension);

[0063] Mask R-CNN instance segmentation and weighted K-means clustering are used to extract multi-entity control points from the input image I, and Gaussian spatial weights are combined to generate a set of control points covering the key motion area.

[0064] Step S1 specifically includes the following steps:

[0065] S11, the original image Input to the pre-trained Mask R-CNN instance segmentation network to generate a set of multi-entity instance masks Among them, H is the image height, W is the image width; 3 is the RGB three-channel; entity instance mask element represents the pixel-level segmentation result of the i-th entity in the j-th frame, where N is the total number of detected entities;

[0066] The original image Input to the pre-trained Mask R-CNN instance segmentation network to generate a multi-entity instance mask set. The specific steps are as follows:

[0067] S111, the original image Input to the pre-trained MaskR-CNN backbone network, and extract multi-scale feature maps through convolutional layers and feature pyramid networks (FPN) Where l represents the feature pyramid level; the backbone network of the pre-trained Mask R-CNN model is ResNet-101-FPN, and the downsampling ratio is 2 l ;

[0068] S112, based on the feature pyramid, the region proposal network (RPN) generates candidate regions through a sliding window The overlapping areas are filtered by non-maximum suppression (NMS), and the top N largest candidate areas are retained; where K is the initial number of candidates, in this embodiment, K is 2000 and N is 1000; (x n ,y n ) is the center coordinate of the candidate region, (wn ,h n ) is the width and height of the candidate region; the threshold value θ of non-maximum suppression NMS Take 0.7;

[0069] The Mask R-CNN network adds a mask prediction branch and ROIAlign layer based on Faster R-CNN, and uses ResNet+FPN as the backbone network for feature extraction, realizing multi-task processing of target detection, classification and pixel-level segmentation; Mask R-CNN mainly consists of three main sub-networks: backbone network, RPN network and head network.

[0070] The Backbone network is used for feature extraction of images. The Head network consists of two main branches: one for classification and bounding box regression, and the other for generating segmentation masks.

[0071] Region Proposal Network (RPN) is a neural network structure used for target detection tasks. Its main function is to quickly generate a set of potential target region proposals in the input image (original image). These regions may be candidate boxes (candidate regions) containing the target.

[0072] The main features and working principles of RPN include:

[0073] Shared convolutional features: RPN and the backbone network of the target detection network share the same set of convolutional feature maps, which can reduce repeated calculations and improve efficiency.

[0074] Sliding window mechanism: RPN generates multiple anchors of fixed scale and size at each position by applying a sliding window on the convolutional feature map. These anchors serve as potential candidate regions.

[0075] Two-classification task: For each anchor point, RPN performs a two-classification task, that is, determining whether the anchor point is foreground (including target) or background.

[0076] Bounding Box Regression: In addition to the classification task, RPN also performs bounding box regression to adjust the position and size of the anchor points so that they more tightly surround the target object.

[0077] Non-maximum suppression (NMS): The generated candidate regions usually overlap. RPN uses a non-maximum suppression strategy to remove redundant candidate regions and retain the regions most likely to contain the target.

[0078] End-to-end training: RPN and the target detection network are trained end-to-end together, and the network parameters are optimized through back-propagation.

[0079] Non-Maximum Suppression (NMS) is an algorithmic step used in object detection. It is primarily used to remove redundant detection results and retain the most likely object boxes. In object detection tasks, multiple candidate boxes (candidate regions) are typically obtained. These candidate boxes may overlap or be very close, so a method is needed to select the best box.

[0080] The basic idea of ​​non-maximum suppression is as follows:

[0081] Sorting: First, all candidate boxes are sorted according to their scores, usually based on classification confidence or other relevant score indicators.

[0082] Select the highest scoring box: Select the candidate box with the highest score as the base box.

[0083] Calculate the intersection over union (IoU): For each remaining candidate box, calculate the ratio of its overlapping part with the current reference box to the sum of the areas of the two boxes (i.e., the intersection over union (IoU)).

[0084] Suppress low-scoring boxes: If the IoU between a candidate box and the reference box exceeds the set threshold θ NMS , they are considered to overlap too much, and the boxes with lower scores are removed.

[0085] Iterative processing: Take the next highest-scoring unremoved box as the new reference box and repeat the above process until all boxes have been processed.

[0086] S113, use the RoIAlign layer to map the N largest candidate regions to the multi-scale feature map, and extract the fixed-size regional features F roi Input to the fully connected classification branch and regression branch, and then filter twice by NMS to obtain the final entity candidate frame set.

[0087] The classification branch is used to output entity category probabilities The regression branch is used to output the bounding box offset The fixed size is 7×7×256, and the regional features The threshold θ of the quadratic NMS NMS Take 0.5; C class is the number of categories in the pre-training dataset;

[0088] S114. For each final entity candidate box set The segmentation head generates a binary mask through a deconvolution layer and pixel-wise Sigmoid activation Then upsample to the original resolution H×W by bilinear interpolation to generate an instance-level mask aligned with the input image

[0089] The RoIAlign (Region of Interest Align) layer is an important component in Mask R-CNN, which is used to accurately map the candidate region (Region of Interest, RoI) from the original image space to the corresponding area on the feature map. This is a prior art and will not be described here.

[0090] S12. Mask elements for each entity instance Calculate its effective pixel area And dynamically determine the number of control points based on area ratio And when the rate of change of the entity mask area Forced setting Where α is the empirical parameter of the density of equilibrium control points; Indicates taking The largest element value in Indicates taking The smallest element value in ; Represents the binary instance mask matrix of the i-th entity in the j-th frame, whose value is defined as: when the pixel coordinate (x, y) belongs to the entity segmentation area Otherwise it is 0, which is used to identify the pixel-level coverage of the entity in the image.

[0091] In this embodiment, α is set to 50, and it can also be modified according to actual needs; when the rate of change of the dynamic area of ​​the video (such as the optical flow gradient) exceeds the threshold (10), setting ≥3 control points can construct a quadratic or higher polynomial trajectory, and by increasing the density of sampling points and the flexibility of the curve, it ensures accurate parameterized modeling of high dynamic motion such as rapid deformation and rotation, and avoids motion distortion caused by underfitting. Therefore, when the rate of change of the entity mask area is Forced setting To ensure dynamic motion capture capabilities;

[0092] S13, effective pixel coordinate set for mask Perform weighted K-means clustering and output the cluster center set after iterative convergence Finally, a set of multi-scale control points covering the key motion area is generated Where N is the total number of entities, Represents the 2D control point coordinate set of the i-th entity in the j-th frame; the weight function of the weighted K-means clustering adopts spatial Gaussian distribution Among them, μ is the centroid of the mask, Controlling the spatial attenuation rate; the key motion areas include edges, joints, significant deformation areas and dynamic texture areas.

[0093] Multi-scale control point set It must be evenly distributed on the surface of the entity and cover key motion areas such as edges and joints, serving as the basic input for subsequent 3D trajectory construction. The multi-scale control point set C is a 2D control point set.

[0094] The weighted K-means clustering algorithm is an improved version of the K-means clustering algorithm that introduces weights to account for the influence of different samples in a dataset on the clustering results. In traditional K-means clustering, the distance between each sample point and the cluster center is equally weighted. In weighted K-means clustering, different sample points can be assigned different weights based on their importance or reliability.

[0095] The steps of weighted K-means clustering generally include:

[0096] Initialize cluster centers and weights: First, select some initial cluster centers and assign an initial weight to each sample point.

[0097] Calculate weighted distance: For each sample point, calculate its weighted distance to each cluster center. The weighted distance can be calculated as a simple product (weight multiplied by distance) or a more complex function.

[0098] Update cluster centers and weights: Based on the weighted distance results, assign each sample point to the nearest cluster center and update the location and weight of the cluster center. This step may involve recalculating the mean of the cluster center and adjusting the weight of the sample point.

[0099] Iterative optimization: Repeat the above steps until the cluster center no longer changes significantly or reaches the preset convergence condition.

[0100] Weighted K-means clustering has applications in many fields, such as machine learning, data analysis, image processing, etc. It can effectively process sample points with different importance, thereby improving the accuracy and robustness of clustering.

[0101] In step S13, the effective pixel coordinate set of the mask Perform weighted K-means clustering and output the cluster center set after iterative convergence The specific steps include:

[0102] S131, based on each entity mask The valid pixel coordinate set When initializing the cluster center, a density-driven strategy is adopted. First, the probability density function of the pixel coordinate space distribution is calculated. Screening initial centroids by kernel density estimation in is the kernel bandwidth parameter; is the local density maximum point; x′, y′ represents the normalized coordinate value after geometric correction or spatiotemporal transformation (such as the new coordinate obtained by affine transformation or optical flow estimation), which is used for feature alignment or motion trajectory correction. Its range is usually [-1, 1] or [0, 1], depending on the normalization method after transformation;

[0103] Kernel Density Estimation (KDE) is a nonparametric statistical method used to estimate the probability density function. It is a method for estimating the probability density function of a random variable. This is a well-known technique and will not be further described here.

[0104] S132, in the iterative optimization stage, the assignment step of the tth iteration assigns each pixel (x, y) to the nearest neighbor cluster center Its distance metric combines spatial coordinates with Gaussian weights:

[0105]

[0106] The weight function is the mask centroid, Control weight decay range;

[0107] The update step recalculates the weighted centroid of each cluster:

[0108]

[0109] in is the pixel set of the kth cluster;

[0110] The iteration termination condition is the centroid offset Or reach the maximum number of iterations T max ;

[0111] Final output cluster center set Where "final" represents the value of t at the end of the iteration, ensuring uniform coverage of the entity's surface features within the mask space, with preferential distribution over areas with high edge curvature and motion-sensitive features (such as limb joints and turning points in object contours). In addition to edges and joints, key motion regions also include areas of significant deformation (such as the curved parts of non-rigid objects) and dynamic textures (such as fabric wrinkles and fluid surface fluctuations). This additional coverage is particularly necessary for non-rigid entities or complex motion scenes.

[0112] In this example, Pixels, maximum number of iterations T max =100;

[0113] S2. Extract the image depth map D through the depth estimation network based on the multi-scale control point set C, and map the multi-scale control point set C combined with the depth value into a global 3D trajectory set T;

[0114] Step S2 specifically includes the following steps:

[0115] S21, set the multi-scale control points Input to the depth estimation network DepthAnythingV2, through its encoder-decoder structure Extract the dense depth map D of the original image I; where N is the total number of entities, Represents the 2D control point coordinate set of the i-th entity in the j-th frame;

[0116] The encoder-decoder structure is Used to extract the dense depth map D of the original image I;

[0117] Step S21 specifically includes the following steps:

[0118] S211, extract the original image through the encoder Multi-scale feature maps Among them D l ={256,512,1024,2048} corresponds to the number of feature channels at different levels. The encoder is the VisionTransformer encoder, version is ViT-L / 16; the downsampling ratio is 2 l ; l is the number of network layers;

[0119] The encoder is a deep learning model that has been pre-trained on large datasets (such as ImageNet). These models (such as ResNet and VGG) have learned rich image features through training on large-scale datasets. The encoder can extract multi-level spatial features from the original image. It is then adapted to the video generation task through lightweight fine-tuning (only optimizing some high-level parameters). At the same time, it is combined with data augmentation in the temporal dimension (such as random frame sampling and time flipping) and a hybrid loss (reconstruction loss + adversarial loss + temporal consistency regularization term) for joint training to ensure that the feature expression takes into account both static semantics and dynamic relevance.

[0120] The depth estimation network DepthAnythingV2 uses a pre-trained Vision Transformer (ViT-L / 16) encoder to extract multi-scale feature maps of the input image. Here, Vision Transformer (ViT-L / 16) is a component in the DepthAnythingV2 network responsible for feature extraction.

[0121] Vision Transformer (ViT-L / 16): The Vision Transformer is a deep learning model for image classification. It's based on the Transformer architecture and was originally developed for natural language processing. The "L" in ViT-L / 16 stands for "large," and the "16" indicates that the input image is segmented into 16x16 patches. This pre-trained model is able to capture high-level semantic information in images.

[0122] DepthAnythingV2 is a network for depth estimation that uses the pre-trained ViT-L / 16 as its feature extractor. The goal of depth estimation is to predict the depth information of each pixel in an image from a single or multiple images, that is, to recover three-dimensional spatial information from a single two-dimensional image.

[0123] The DepthAnythingV2 network uses ViT-L / 16 to process input images. ViT-L / 16 divides the input image into fixed-size patches and extracts features through its Transformer architecture. These features contain multi-scale information about the image, which aids in subsequent depth estimation.

[0124] Different layers of ViT-L / 16 output feature maps of different scales. Lower layers may capture detailed information, while higher layers capture more abstract semantic information. These multi-scale feature maps are important for depth estimation tasks because different objects and structures in a scene may appear at different scales.

[0125] Downsampling refers to reducing the resolution of the feature map. In ViT-L / 16, as the number of network layers increases, the resolution of the feature map gradually decreases, which is usually achieved through convolution operations between layers. The downsampling ratio refers to the reduction ratio of the feature map relative to the original image. For example, if the original image size is 256x256 and the feature map size is 64x64, then the downsampling ratio is 4;

[0126] S212, the decoder gradually upsamples the feature map through the cascade transposed convolution module to obtain enhanced features The regression head enhances the features through convolution operation and Sigmoid activation function Mapped to the dense depth map D of the original image I, satisfying in is a learnable parameter, σ is the Sigmoid activation function; D(x,y) represents the normalized relative depth value of the dense depth map D at the pixel coordinate (x,y), with a value range of [0,1], where 0 represents the nearest and 1 represents the farthest;

[0127] The decoder gradually upsamples the feature map through cascaded transposed convolution modules. Specifically, each transposed convolution module first increases the resolution of the feature map by 2 times, concatenates it with the corresponding layer features of the encoder, and then refines it through a 3×3 convolution residual block, and finally outputs the enhanced feature. The regression head enhances the features through convolution operation and Sigmoid activation function Mapping to normalized relative depth map That is, the dense depth map D of the original image I satisfies in is a learnable parameter, σ is the Sigmoid activation function, and the depth value range is [0,1].

[0128] The Cascaded Transposed Convolution Module (CTCM) is a structural design for progressive upsampling in deep learning, commonly used in image generation, super-resolution reconstruction, or segmentation tasks. Its core idea is to gradually recover high-resolution details from low-resolution features by cascading multiple layers of transposed convolution.

[0129] The transposed convolution module belongs to the decoder component. The decoder contains multiple transposed convolution modules. First, the feature map resolution is doubled, and then it is concatenated with the corresponding layer features of the encoder and refined by a 3×3 convolution residual block. Finally, the enhanced features are output. The regression head enhances the features through convolution operation and Sigmoid activation function The ViT-L / 16 encoder extracts global features solely through patch embedding and self-attention layers, while the transposed convolution module, located on the decoder side (e.g., the U-Net architecture), upsamples the low-resolution feature maps by a factor of 2 (restoring spatial detail) and fuses them with the encoder's skip-connected features to achieve a step-by-step resolution increase. The encoder performs dimensionality reduction to extract semantic information, while the decoder performs transposed convolution to reconstruct a high-resolution output.

[0130] S22, use the bilinear interpolation formula for each control point Assigning Depth Values Then construct the 3D trajectory point of the i-th entity in the j-th frame And through the timing smoothing constraint Eliminate depth jump noise and finally output a global 3D trajectory set in represents the 3D trajectory matrix of the i-th entity in F frames; F is the total number of video frames;

[0131] The bilinear interpolation formula is shown below:

[0132]

[0133] Among them, w(m,n) is the interpolation weight function; D(m,n) represents the discrete sampling value of the density map D at the integer coordinate (m,n) (such as the pixel value of the mth row and nth column of the feature map), and D(x,y) is the estimated value at the continuous coordinate (x,y) calculated by bilinear interpolation, which is obtained by the weighted average of the D(m,n) values ​​of the four adjacent grid points (m,n), (m+1,n), (m,n+1), and (m+1,n+1). The weight is determined by the decimal part of x and y, thereby achieving a smooth transition from discrete to continuous.

[0134] This step builds multi-entity 3D trajectories based on the DepthAnythingV2 depth estimation network and integrates temporal smoothing constraints to eliminate depth jump noise.

[0135] S3, use discrete wavelet transform to perform time-frequency decomposition on the global 3D trajectory set T to obtain low-frequency approximate components and high-frequency detail components, combined with the user direction command U and the adaptive direction gain Adjust the high-frequency detail components to optimize the global 3D trajectory set T to obtain the optimized 3D trajectory set

[0136] Step S3 specifically includes the following steps:

[0137] S31, the global 3D trajectory collection Timing direction control instructions input by the user The signals are input to the time-frequency analysis module, and the trajectory signal is decomposed into multiple scales by discrete wavelet transform to obtain the low-frequency approximate component. and high-frequency detail components satisfy in represents the 3D trajectory matrix of the i-th entity in the F frame; is the unit direction vector of the i-th entity in the j-th frame; L is the number of decomposition levels, the default is L = 3;

[0138] Specifically, the multi-scale decomposition of the trajectory signal by discrete wavelet transform is as follows: for each entity trajectory T i The three-dimensional components of are decomposed by applying Daubechies-4 wavelet basis function ψ(t); the three-dimensional components include X axis, Y axis and Z axis;

[0139] Discrete Wavelet Transform (DWT) is an important time-frequency analysis tool that decomposes the signal into wavelet components of different scales (frequencies);

[0140] Daubechies-4 is a specific wavelet basis function proposed by Ingrid Daubechies. It is one of the Daubechies wavelet series.

[0141] The time-frequency analysis module consists of three parts: a discrete wavelet transform (DWT) decomposition layer, a directional gain adjustment unit, and a signal reconstruction algorithm. It decomposes the trajectory signal into a low-frequency approximate component (representing the global motion trend) and a high-frequency detail component (representing local motion noise and details) using wavelet basis functions (such as Daubechies-4). It then performs adaptive directional weighting on the high-frequency component based on user directional instructions (such as enhancing the motion amplitude in a specific direction and suppressing irrelevant noise). Finally, it reconstructs the optimized trajectory signal through an inverse wavelet transform, achieving joint time-frequency optimization of the motion path.

[0142] S32, based on the user direction instruction U, the high frequency detail component Perform directional weighted adjustment, specifically: for the lth layer detail component Apply adaptive directional gain At the same time, through the threshold function Suppress noise and reconstruct optimized trajectory The final output is the optimized 3D trajectory set after time-frequency optimization Among them, ⊙ represents element-by-element multiplication; τ l =0.05×2 l-L ; is the layer-dependent gain coefficient, λ l =0.1×2 L-l ;λ l It is used to balance the loss contributions of different decoder levels (such as shallow details and deep semantics). The value usually decreases as the level deepens (such as shallow λ = 1.0, deep λ = 0.2). It is determined by grid search or heuristic rules based on feature sensitivity to optimize the multi-scale feature alignment effect.

[0143] S4: Input the multi-entity instance mask set M, the global 3D trajectory set T, and the optimized 3D trajectory set T′ into the multi-scale fusion network to generate the entity-level optical flow field O. i With multi-scale features And the entity-level optical flow field O is fused through the gated cross-scale attention mechanism i With multi-scale features Generate a multi-scale control signal S; the frame-by-frame control signal S integrates multimodal information of entity motion, segmentation and image content;

[0144] Step S4 specifically includes the following steps:

[0145] S41. Mask the entity instance collection Global 3D trajectory collection And optimize the 3D trajectory set T′={T′ i} are jointly input into a multi-scale control feature fusion network; the multi-scale control feature fusion network consists of a cascaded encoder-decoder structure and a gated cross-scale attention module;

[0146] S42, based on each entity trajectory in T' The optical flow generation module calculates the frame-by-frame dense optical flow field O i , the frame-by-frame dense optical flow field O i The value of the optimized 3D trajectory point Projection transformation in camera coordinate system Determine; among them is the camera internal parameter, R∈SO(3), is an external parameter; the optimized 3D trajectory point

[0147] Directly generated based on depth estimation, It integrates time-frequency domain motion constraints and user control intentions.

[0148] represents the 3D trajectory matrix of the i-th entity after optimization in the j-th frame (output in step S3); T′ i is the specific trajectory point (such as position coordinate) in the trajectory matrix. The two are equivalent in symbols and have the same context.

[0149] The optical flow generation module is: in Represents the displacement vector of the i-th entity at pixel (x, y) in the j-th frame;

[0150] S43. Construct a multi-scale feature pyramid: Extract hierarchical features from the original image I through the ResNet-50 backbone network And the optical flow field O i Generate by downsampling to the corresponding scale through bilinear interpolation And in each scale l, the multi-scale features are fused into the hierarchical features through the gated cross-scale attention module

[0151] The formula of the gated cross-scale attention module is:

[0152]

[0153] Where W g,l 、W a,l 、 is the learnable weight matrix, represents the convolution operation, [·] is channel concatenation, and ⊙ is element-by-element multiplication; C l is the number of channels of the lth layer feature map extracted by the ResNet-50 backbone network, and its value is determined by the network structure. l ={256,512,1024,2048}; Represents the downsampled optical flow field of the i-th entity at the l-th scale;

[0154] S44, multi-scale features through skip connection It is concatenated with the decoder upsampled features and fused with convolution kernels to output spatially aligned multi-scale control signals. It satisfies the spatial alignment constraint Where D is the control signal dimension; the convolution kernel is a 3×3 convolution kernel; MLP is a multi-layer perceptron, which performs nonlinear transformation and dimensionality compression on multi-scale features to generate cross-scale gating weights for dynamic fusion of motion control signals at different resolutions; S j (x, y) represents the motion control intensity weight of the j-th frame at the spatial coordinate (x, y), which is calculated by the Softmax normalized optical flow field amplitude and the control point distribution density, and is used to represent the contribution weight of this position to the trajectory optimization;

[0155] In this embodiment, D=64;

[0156] S5. Input the multi-scale control signal S and the original image I into the improved StableVideoDiffusion model. In the latent space diffusion process of the improved StableVideoDiffusion model, the video potential representation sequence is generated through annealing sampling and cross-modal attention.

[0157] Specifically, the improved StableVideoDiffusion model introduces an annealing sampling strategy and a cross-modal spatiotemporal attention module in the latent space diffusion process. The specific improvements include: embedding a cross-modal attention mechanism in the UNet architecture, and dynamically calculating the latent variable z t The spatiotemporal correlation weight α with the multi-scale control signal conditional embedding c t,j (x,y), achieving fine-grained alignment of control signals with the diffusion process;

[0158] Specifically, the original image I is compressed into a latent space representation z0 through a 3D-VAE encoder, and the frame-by-frame control signal S is mapped into a conditional embedding c; the denoising latent variable z is iteratively denoised through the improved UNet architecture. t , integrating control signals and temporal features, and using annealing strategy to dynamically adjust the control signal weight to balance the generation freedom and constraint strength. The denoised latent variables are optimized for temporal coherence by the frequency domain stabilization module, and finally the 3D-VAE decoder outputs the video potential representation sequence The frequency domain stabilization module includes FFT frequency domain filtering and inverse transformation; where H and W are the frame height and width, and F is the total number of frames;

[0159] The improved Stable Video Diffusion model is a model based on the Stable Video Diffusion framework;

[0160] The Stable Video Diffusion (SVD) framework uses a 2D encoder / decoder by default to process video frame sequences. The improved Stable Video Diffusion model replaces its encoder / decoder with a 3D-VAE (introducing three-dimensional convolutional kernels) to simultaneously model spatial features and temporal continuity, thereby more efficiently extracting and reconstructing the spatiotemporal correlation of videos. Specific improvements include: 1) using 3D-VAE to enhance inter-frame dynamic modeling; 2) incorporating temporal conditional residual blocks and cross-modal attention mechanisms into UNet; and 3) expanding 3D convolutional layers to enhance temporal consistency. These changes optimize the generation quality and control accuracy of highly dynamic motion (such as rapid deformation and multi-entity interaction).

[0161] The diffusion process of the Stable Video Diffusion framework is an iterative denoising method for generating video sequences. It is based on deep learning models, specifically conditional generative models. The following are the general steps of the diffusion process:

[0162] Initialization: First, start with the original video sequence, where the video frames are clear and in order.

[0163] Gradual Noise Addition: During the diffusion process, noise is gradually added to the video frame, usually in multiple time steps. At each step, the amount of noise added gradually increases until the video frame becomes almost indistinguishable from pure random noise.

[0164] Latent representation: At each time step, the mixture of video frame and noise is converted into a latent space representation (usually in vector or matrix form) that captures the content and noise level of the current frame.

[0165] Conditional embedding: During the denoising process, an additional conditional signal (such as text description, keyframes, audio, etc.) may be required. This signal is converted into a conditional embedding through an embedding network to guide the denoising process.

[0166] Denoising iteration: Starting from the maximum noise level, denoising is performed step by step in reverse. At each step, a pre-trained neural network (such as a U-Net architecture) is used to predict the noise at the current noise level and remove it from the latent representation. This network usually accepts the latent representation and conditional embedding of the current time step as input.

[0167] Generating video frames: After denoising for all time steps, the final latent representation is converted back to the video frame domain to generate clear video sequences.

[0168] Post-processing: The generated video may require post-processing, such as adjusting brightness and contrast, to improve the visual effect.

[0169] The key to this process is training a neural network that can accurately predict noise, so that high-quality video content can be gradually restored during the denoising process. In this way, the Stable Video Diffusion framework can generate coherent and realistic video sequences given certain conditions (such as text, audio, or other video frames).

[0170] Step S5 specifically includes the following steps:

[0171] S51, multi-scale control signal With the original image Input to the improved model based on the Stable Video Diffusion framework, first compressing the input image I into a potential representation through the 3D-VAE encoder At the same time, the multi-scale control signal S is mapped into a conditional embedding through the spatiotemporal convolutional network. In the diffusion denoising process, the latent variable z t Through the iterative update of the improved UNet architecture, the time-dependent residual block calculation form is:

[0172]

[0173] Among them, AdaGN(z t ,t) is the embedding vector injected at time step t, representing the adaptive group normalization layer; CrossAttn(z t ,c) is the cross-modal attention module, used to calculate z t c interaction weight And weighted fusion as conditional features; h = H / 8; w = W / 8; d = 4; t∈[1,T]; Represents the output of the denoising network, whose input is the current noise latent variable z t , time step t (control denoising stage) and conditional signal c (such as optical flow trajectory), used to predict the current noise to achieve step-by-step denoising; Conv 3D It is a three-dimensional convolutional layer that jointly models motion continuity in the spatial dimension (H×W) and the temporal dimension (implicit stacking between frames) to ensure the spatiotemporal consistency of the generated results across frames.

[0174] The hidden variable z t It is generated iteratively by the forward diffusion process, by injecting Gaussian noise into the initial latent variable z0 (extracted by the VAE encoder) at each time step t According to the preset noise scheduling β t calculate where α t =1-β t , is α t The cumulative product of

[0175] 3D-VAE (Variational Autoencoder for 3D data) is a variational autoencoder specifically designed for processing 3D data. It consists of two parts: an encoder and a decoder.

[0176] The encoder’s task is to transform high-dimensional 3D input data (such as point clouds, meshes, or voxelized 3D shapes) into a low-dimensional latent space (latent variable) representation.

[0177] The task of the decoder is to reconstruct the original high-dimensional 3D data from the low-dimensional latent space (latent variable) representation.

[0178] The improved UNet architecture contains a time-dependent residual block (through the embedding of conditional parameters of time step t) and a cross-modal spatiotemporal attention module, which together constitute the core of the improvement. The former realizes the temporal coherence modeling of motion dynamics, and the latter completes the fine-grained alignment of control signals and latent variables.

[0179] The improved UNet architecture improves the denoising effect mainly through three aspects: first, the time-dependent residual block is introduced. By embedding the time step t into the parameters of the adaptive normalization layer, the network can dynamically perceive the characteristics of different denoising stages and distinguish the stage tasks of noise elimination and detail reconstruction; secondly, a cross-modal spatiotemporal attention module is designed, which uses the optical flow field and control point trajectory as conditional signals and combines them with the latent variable z tThe algorithm uses cross-modal interaction with queries to achieve refined guidance of motion direction and amplitude. Finally, it expands the 3D convolution kernel in the deep feature fusion layer to explicitly capture inter-frame motion continuity through joint spatiotemporal modeling (spatial and temporal dimensions), avoiding the timing jitter caused by independent frame-by-frame generation. These improvements enhance the model's adaptability to complex motion patterns (such as rapid acceleration and multi-entity interaction) and improve the spatiotemporal consistency of generated results through efficient alignment of conditional signals with latent variables.

[0180] S52, introduce annealing sampling strategy: in the denoising step t∈[T c ,T end ], and in t∈[1,T c ] gradually attenuates the weight of the multi-scale control signal Sγ(t)=min(1,(T c -t) / (T c -1)) to eliminate artifacts caused by over-constraint; annealing from T c To the final step T end ;T end represents the termination time step of the annealing sampling strategy; T c Indicates the starting time step of the annealing sampling strategy, which is usually taken as the last 20% to 30% of the total number of steps of the annealing sampling strategy. For example, if the total number of steps is 1000, T c =800;

[0181] Finally, the potential representation is stabilized by the frequency domain module Post-processing outputs temporally smoothed video latent representation sequence is a learnable frequency domain filter.

[0182] The final potential representation is stabilized by the frequency domain module Post-processing to output temporally smoothed video sequences Specifically: for each frame feature Perform fast Fourier transform to obtain frequency domain components Applying spectral attention mechanism Then restore the time domain features through inverse FFT Finally, the 3D-VAE decoder Output temporally smoothed video latent representation sequence

[0183] S6. Video potential representation sequence Each frame image is divided into local blocks and decomposed into amplitude spectrum M and phase spectrum P through FFT transform. The amplitude spectrum is adaptively smoothed and optimized based on the cross-frame consistency constraint and combined with the phase spectrum P. The inverse FFT is used to restore the time domain image block. The complete video frame is reconstructed by the weighted overlap method to output the final video sequence X′final .

[0184] Using frequency domain stabilization module to stabilise video latent representation sequence The time-frequency energy consistency constraint is used to suppress flicker artifacts. Enhance frequency domain stability. Adaptively smooth and optimize the amplitude spectrum based on cross-frame consistency constraints. The amplitude spectrum M is used to characterize frequency domain energy.

[0185] Step S6 specifically includes the following steps:

[0186] S61, video potential representation sequence Input to the frequency domain stabilization module, the video flicker and motion artifacts are eliminated by time-frequency joint, and then the video potential representation sequence is converted into Each frame image is divided into local blocks and decomposed into amplitude spectrum M and phase spectrum P through FFT transformation;

[0187] Specifically, first, for each frame of image Perform window overlapping block processing to obtain a set of local image blocks For each image block B j,m,n Perform two-dimensional fast Fourier transform (2D-FFT) to obtain frequency domain components And calculate its amplitude spectrum M j,m,n (u,v)=‖F j,m,n (u,v)‖2 and phase spectrum P j,m,n (u,v)=∠F j,m,n (u,v); where represents the generated image of the jth frame; the window size k of the window overlapping block processing is 32, the window overlap rate is 50%, and the step size S is 16;

[0188] The unit of size in this embodiment is pixel, which can also be set according to actual needs.

[0189] Each frame is evenly divided into W×W windows (e.g., W = 32 pixels). Adjacent windows are overlapped horizontally and vertically with a sliding step size of S = WP (S = 24 in this example). This ensures that the windows cover the entire image and that overlapping areas are fused using a weighted average to avoid boundary artifacts. After partitioning, each window is independently fed into an encoder to extract local features. The resulting images are then sequentially assembled into a complete feature map for subsequent fine-tuning of the motion trajectory.

[0190] S62. Based on the cross-frame frequency domain consistency constraint, the amplitude spectrum is adaptively smoothed: the neighborhood mean is calculated using a sliding window, and the local and global frequency domain features are fused through learnable spectral attention weights to generate an optimized amplitude spectrum M′ that suppresses high-frequency noise. The optimized amplitude spectrum M′ is combined with the phase spectrum P, and the time domain image block is restored through inverse FFT. The complete video frame is reconstructed through the weighted overlap method to output the final video sequence X′. final .

[0191] Specifically, through the sliding window Calculate neighborhood mean And use parameterized spectral attention weights Fusion of local and global frequency domain features to obtain optimized amplitude spectrum The optimized amplitude spectrum M′ j,m,n Compared with the original phase spectrum P j,m,n Combination Recover the time domain image block B′ by inverse FFT j,m,n =IFFT(F′ j,m,n ), and reconstruct the complete frame X' using weighted overlap-add method j , and finally output the final video sequence with frequency domain smoothing

[0192] in, is a learnable parameter, T represents the length of the temporal context window used in the motion trajectory optimization process (for example, T = 3 means that the adjacent temporal information of the current frame and the frames before and after it are considered simultaneously), which is used to constrain the trajectory smoothness through local temporal correlation; in this embodiment, T = 3;

[0193] Weighted overlap-add (WOLA) is a signal processing technique used to process a signal (such as an audio signal or image frame) in blocks and then reassemble the blocks to recover the original continuous signal. WOLA is an extension of the overlap-add method. It introduces weighting factors when reassembling blocks to improve the smoothness of transitions at block edges. This is a prior art technique and will not be further described here.

[0194] This embodiment optimizes 3D trajectories through multi-entity segmentation, depth estimation, and time-frequency decomposition, combined with user instructions, and uses a multi-scale fusion network to generate control signals. Finally, these signals are input into the improved Stable Video Diffusion model with the original image to generate a video latent representation sequence. This solves the problems of insufficient precision in dynamic entity motion control and poor cross-frame consistency in existing video generation methods. Through the depth-guided 3D trajectory modeling and time-frequency joint optimization mechanism, the motion smoothness, spatial realism, and time-frequency stability of the generated video are significantly improved.

[0195] Example 2

[0196] refer to Figure 2 This embodiment provides a trajectory control video generation device based on depth information and time-frequency optimization, including the following units:

[0197] A control point generation unit is configured to perform multi-entity instance segmentation on the original image I to generate a multi-entity instance mask set M, dynamically determine the number of control points based on the area of ​​the entity region, and extract a multi-scale control point set C covering the key motion region through weighted clustering; the multi-scale control point set C is a 2D control point set;

[0198] A global 3D trajectory generation unit is used to extract an image depth map D based on a multi-scale control point set C through a depth estimation network, and map the multi-scale control point set C combined with the depth value into a global 3D trajectory set T;

[0199] 3D trajectory optimization unit, used to use discrete wavelet transform to perform time-frequency decomposition on the global 3D trajectory set T to obtain low-frequency approximate components and high-frequency detail components, combined with the user direction instruction U and the adaptive direction gain Adjust the high-frequency detail components to optimize the global 3D trajectory set T to obtain the optimized 3D trajectory set T′;

[0200] The multi-scale control signal generation unit is used to input the multi-entity instance mask set M, the global 3D trajectory set T and the optimized 3D trajectory set T′ into the multi-scale fusion network to generate the entity-level optical flow field O i With multi-scale features And the entity-level optical flow field O is fused through the gated cross-scale attention mechanism i With multi-scale features Generate a multi-scale control signal S;

[0201] The video potential representation sequence generation unit is used to input the multi-scale control signal S and the original image I into the improved Stable Video Diffusion model. The video potential representation sequence is generated by annealing sampling and cross-modal attention in the latent space diffusion process of the improved Stable Video Diffusion model.

[0202] Video denoising unit, used to transform the video potential representation sequence Each frame image is divided into local blocks and decomposed into amplitude spectrum M and phase spectrum P through FFT transform. The amplitude spectrum is adaptively smoothed and optimized based on the cross-frame consistency constraint and combined with the phase spectrum P. The inverse FFT is used to restore the time domain image block. The complete video frame is reconstructed by the weighted overlap method to output the final video sequence X′ final .

[0203] Example 3

[0204] refer to Figure 3 , Figure 3 is a schematic diagram of the structure of a trajectory control video generation device based on depth information and time-frequency optimization according to this embodiment. The trajectory control video generation device 20 based on depth information and time-frequency optimization according to this embodiment includes a processor 21, a memory 22, and a computer program stored in the memory 22 and executable by the processor 21. When the processor 21 executes the computer program, the steps of the aforementioned method embodiments are implemented. Alternatively, when the processor 21 executes the computer program, the functions of the modules / units in the aforementioned apparatus embodiments are implemented.

[0205] Exemplarily, the computer program can be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to implement the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution process of the computer program in the trajectory control video generation device 20 based on depth information and time-frequency optimization. For example, the computer program can be divided into the modules described in Example 2. For the specific functions of each module, please refer to the operating process of the device described in the above embodiment, and will not be repeated here.

[0206] The trajectory control video generation device 20 based on depth information and time-frequency optimization may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will appreciate that the schematic diagram is merely an example of the trajectory control video generation device 20 based on depth information and time-frequency optimization and does not limit the trajectory control video generation device 20 based on depth information and time-frequency optimization. The device 20 may include more or fewer components than shown in the figure, or may combine certain components, or different components. For example, the trajectory control video generation device 20 based on depth information and time-frequency optimization may further include input and output devices, network access devices, buses, and the like.

[0207] The processor 21 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 21 is the control center of the trajectory control video generation device 20 based on depth information and time-frequency optimization, and utilizes various interfaces and lines to connect various parts of the entire trajectory control video generation device 20 based on depth information and time-frequency optimization.

[0208] The memory 22 can be used to store the computer programs and / or modules. The processor 21 implements the various functions of the trajectory control video generation device 20 based on depth information and time-frequency optimization by running or executing the computer programs and / or modules stored in the memory 22 and calling the data stored in the memory 22. The memory 22 can mainly include a program storage area and a data storage area. The program storage area can store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created based on the use of the mobile phone (such as audio data, a phone book, etc.). In addition, the memory 22 can include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0209] Wherein, if the module / unit integrated in the trajectory control video generation device 20 based on depth information and time-frequency optimization is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor 21, it can implement the steps of the above-mentioned various method embodiments. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practices in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practices, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0210] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art can understand and implement the present invention without inventive effort.

[0211] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 Multiple processes and / or boxes Figure 1 A device that performs the functions specified in one or more boxes.

[0212] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0213] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0214] The parts of the present invention that are not described in detail are prior art. It is obvious to those skilled in the art that the present invention is not limited to the details of the above-mentioned exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive, and it is intended that all changes that fall within the meaning and scope of equivalent elements are included in the present invention.

Claims

1. A trajectory control video generation method based on depth information and time-frequency optimization, characterized in that: The following steps are involved: S1. Perform multi-entity instance segmentation on the original image I to generate a multi-entity instance mask set M, and dynamically determine the number of control points based on the area of ​​the entity region to extract a multi-scale control point set C covering the key motion area through weighted clustering; the multi-scale control point set C is a 2D control point set; S2. Extract the image depth map D through the depth estimation network based on the multi-scale control point set C, and map the multi-scale control point set C combined with the depth value into a global 3D trajectory set T; S3, use discrete wavelet transform to perform time-frequency decomposition on the global 3D trajectory set T to obtain low-frequency approximate components and high-frequency detail components, combined with the user direction command U and the adaptive direction gain Adjust the high-frequency detail components to optimize the global 3D trajectory set T to obtain the optimized 3D trajectory set T′; S4: Input the multi-entity instance mask set M, the global 3D trajectory set T, and the optimized 3D trajectory set T′ into the multi-scale fusion network to generate the entity-level optical flow field O. i With multi-scale features And the entity-level optical flow field O is fused through the gated cross-scale attention mechanism i With multi-scale features Generate a multi-scale control signal S; S5. Input the multi-scale control signal S and the original image I into the improved Stable Video Diffusion model. In the latent space diffusion process of the improved Stable Video Diffusion model, the video potential representation sequence is generated through annealing sampling and cross-modal attention. Step S5 specifically includes the following steps: S51, multi-scale control signal With the original image Input to the improved model based on the Stable Video Diffusion framework, first compressing the input image I into a potential representation through the 3D-VAE encoder At the same time, the multi-scale control signal S is mapped into a conditional embedding through the spatiotemporal convolutional network. In the diffusion denoising process, the latent variable z t Through the iterative update of the improved UNet architecture, its time-dependent residual block calculation form is: ò θ (z t ,t,c)=Conv 3D (AdaGN(z t ,t)+CrossAttn(z t ,c)), Among them, AdaGN(z t ,t) is the embedding vector injected at time step t, representing the adaptive group normalization layer; CrossAttn(z t ,c) is the cross-modal attention module, used to calculate z t c interaction weight And weighted fusion as conditional features; h = H / 8; w = W / 8; d = 4; t∈[1,T]; θ (z t ,t,c) represents the output of the denoising network; Conv 3D is a 3D convolutional layer that jointly models motion continuity in both spatial and temporal dimensions; H is the image height and W is the image width; S52, introduce annealing sampling strategy: in the denoising step t∈[T c ,T end ], and in t∈[1,T c ] gradually attenuates the weight of the multi-scale control signal Sγ(t)=min(1,(T c -t) / (T c -1)) to eliminate artifacts caused by over-constraints; finally, the potential representation is stabilized by the frequency domain stabilization module Post-processing outputs temporally smoothed video latent representation sequence is a learnable frequency domain filter; T end represents the termination time step of the annealing sampling strategy; T c Indicates the starting time step of the annealing sampling strategy.

2. The trajectory control video generation method based on depth information and time-frequency optimization according to claim 1 is characterized in that: Step S1 specifically includes the following steps: S11, the original image Input to the pre-trained Mask R-CNN instance segmentation network to generate a set of multi-entity instance masks Among them, H is the image height, W is the image width; 3 is the RGB three-channel; entity instance mask element represents the pixel-level segmentation result of the i-th entity in the j-th frame, where i∈[1,N] represents the entity index and N is the total number of entities; S12. Mask elements for each entity instance Calculate its effective pixel area And dynamically determine the number of control points based on area ratio And when the rate of change of the entity mask area Forced setting Where α is the empirical parameter of the density of equilibrium control points; Indicates taking The largest element value in Indicates taking The smallest element value in ; represents the binary instance mask matrix of the i-th entity in the j-th frame; S13, effective pixel coordinate set for mask Perform weighted K-means clustering and output the cluster center set after iterative convergence Finally, a set of multi-scale control points covering the key motion area is generated Where N is the total number of entities, Represents the 2D control point coordinate set of the i-th entity in the j-th frame; the weight function of the weighted K-means clustering adopts spatial Gaussian distribution Among them, μ is the mask centroid, Controlling the spatial attenuation rate; the key motion areas include edges, joints, significant deformation areas and dynamic texture areas.

3. The trajectory control video generation method based on depth information and time-frequency optimization according to claim 1 is characterized in that: Step S2 specifically includes the following steps: S21, set the multi-scale control points Input to the depth estimation network DepthAnythingV2, through its encoder-decoder structure Extract the dense depth map D of the original image I; where N is the total number of entities; Represents the 2D control point coordinate set of the i-th entity in the j-th frame; S22, use the bilinear interpolation formula for each control point Assigning Depth Values Then construct the 3D trajectory point of the i-th entity in the j-th frame And through the timing smoothing constraint Eliminate depth jump noise and finally output a global 3D trajectory set in Represents the 3D trajectory matrix of the i-th entity in F frames; F is the total number of video frames.

4. The trajectory control video generation method based on depth information and time-frequency optimization according to claim 2 is characterized in that: Step S3 specifically includes the following steps: S31, the global 3D trajectory collection Timing direction control instructions input by the user The signals are input to the time-frequency analysis module, and the trajectory signal is decomposed into multiple scales by discrete wavelet transform to obtain the low-frequency approximate component. and high-frequency detail components satisfy in represents the 3D trajectory matrix of the i-th entity in the F frame; is the unit direction vector of the i-th entity in the j-th frame; L is the number of decomposition layers; S32, based on the timing direction control instruction U, the high frequency detail component Perform directional weighted adjustment, specifically: for the lth layer detail component Apply adaptive directional gain At the same time, through the threshold function Suppress noise and reconstruct optimized trajectory The final output is the optimized 3D trajectory set after time-frequency optimization Among them, ⊙ represents element-by-element multiplication, is the layer-dependent gain coefficient.

5. The trajectory control video generation method based on depth information and time-frequency optimization according to claim 1 is characterized in that: Step S4 specifically includes the following steps: S41. Mask the entity instance collection Global 3D trajectory collection And optimize the 3D trajectory set T′={T′ i } are jointly input into a multi-scale control feature fusion network; the multi-scale control feature fusion network consists of a cascaded encoder-decoder structure and a gated cross-scale attention module; S42, based on each entity trajectory in T' The optical flow generation module calculates the frame-by-frame dense optical flow field O i , the frame-by-frame dense optical flow field O i The value of the optimized 3D trajectory point Projection transformation in camera coordinate system Determine; among them is the camera internal parameter, R∈SO(3), is an external parameter; the optimized 3D trajectory point S43. Construct a multi-scale feature pyramid: Extract hierarchical features from the original image I through the ResNet-50 backbone network And the optical flow field O i Generate by downsampling to the corresponding scale through bilinear interpolation And in each scale l, the multi-scale features are fused into the hierarchical features through the gated cross-scale attention module The formula of the gated cross-scale attention module is as follows: in is the learnable weight matrix, represents the convolution operation, [·] is channel concatenation, and ⊙ is element-by-element multiplication; C l is the number of channels of the l-th layer feature map extracted by the ResNet-50 backbone network, and its value is determined by the network structure; Represents the downsampled optical flow field of the i-th entity at the l-th scale; S44, multi-scale features through skip connection It is concatenated with the decoder upsampled features and fused with convolution kernels to output spatially aligned multi-scale control signals. It satisfies the spatial alignment constraint Where D is the control signal dimension; the convolution kernel is a 3×3 convolution kernel; MLP is a multi-layer perceptron; S j (x,y) represents the motion control strength weight of the j-th frame at the spatial coordinate (x,y).

6. The trajectory control video generation method based on depth information and time-frequency optimization according to claim 1 is characterized in that: The method further comprises: S6. Video potential representation sequence Each frame image is divided into local blocks and decomposed into amplitude spectrum M and phase spectrum P through FFT transform. The amplitude spectrum is adaptively smoothed and optimized based on the cross-frame consistency constraint and combined with the phase spectrum P. The inverse FFT is used to restore the time domain image block. The complete video frame is reconstructed by the weighted overlap method to output the final video sequence X′ final .

7. The method for generating trajectory control video based on depth information and time-frequency optimization according to claim 6, characterized in that: Step S6 specifically includes the following steps: S61, video potential representation sequence Input to the frequency domain stabilization module, the video flicker and motion artifacts are eliminated by time-frequency joint, and then the video potential representation sequence is converted into Each frame image is divided into local blocks and decomposed into amplitude spectrum M and phase spectrum P through FFT transformation; S62. Based on the cross-frame frequency domain consistency constraint, the amplitude spectrum is adaptively smoothed: the neighborhood mean is calculated using a sliding window, and the local and global frequency domain features are fused through learnable spectral attention weights to generate an optimized amplitude spectrum M′ that suppresses high-frequency noise. The optimized amplitude spectrum M′ is combined with the phase spectrum P, and the time domain image block is restored through inverse FFT. The complete video frame is reconstructed through the weighted overlap method to output the final video sequence X′. final .

8. The method for generating trajectory control video based on depth information and time-frequency optimization according to claim 2, characterized in that: Step S11 specifically includes the following steps: S111, the original image Input to the pre-trained Mask R-CNN backbone network, and extract multi-scale feature maps through convolutional layers and feature pyramid networks Where l represents the feature pyramid level; the backbone network of the pre-trained Mask R-CNN model is ResNet-101-FPN, and the downsampling ratio is 2 l ; S112, based on the feature pyramid, the region proposal network generates candidate regions through a sliding window The overlapping areas are filtered by non-maximum suppression, and the top N largest candidate areas are retained; where K is the initial number of candidates, (x n ,y n ) is the center coordinate of the candidate region, (w n ,h n ) is the width and height of the candidate region; the threshold value θ of non-maximum suppression NMS Take 0.7; S113, use the RoIAlign layer to map the N largest candidate regions to the multi-scale feature map, and extract the fixed-size regional features F roi Input to the fully connected classification branch and regression branch, and then filter twice by NMS to obtain the final entity candidate frame set. S114. For each final entity candidate box set The segmentation head generates a binary mask through a deconvolution layer and pixel-wise Sigmoid activation Then upsample to the original resolution H×W by bilinear interpolation to generate a set of multi-entity instance masks aligned with the input image 9. A trajectory control video generation device based on depth information and time-frequency optimization, characterized in that: Includes the following units: A control point generation unit is configured to perform multi-entity instance segmentation on the original image I to generate a multi-entity instance mask set M, dynamically determine the number of control points based on the area of ​​the entity region, and extract a multi-scale control point set C covering the key motion region through weighted clustering; the multi-scale control point set C is a 2D control point set; The global 3D trajectory generation unit extracts the image depth map D based on the multi-scale control point set C through the depth estimation network, and maps the multi-scale control point set C combined with the depth value into the global 3D trajectory set T; 3D trajectory optimization unit, used to use discrete wavelet transform to perform time-frequency decomposition on the global 3D trajectory set T to obtain low-frequency approximate components and high-frequency detail components, combined with the user direction instruction U and the adaptive direction gain Adjust the high-frequency detail components to optimize the global 3D trajectory set T to obtain the optimized 3D trajectory set T′; The multi-scale control signal generation unit is used to input the multi-entity instance mask set M, the global 3D trajectory set T and the optimized 3D trajectory set T′ into the multi-scale fusion network to generate the entity-level optical flow field O i With multi-scale features And the entity-level optical flow field O is fused through the gated cross-scale attention mechanism i With multi-scale features Generate a multi-scale control signal S; The video potential representation sequence generation unit is used to input the multi-scale control signal S and the original image I into the improved Stable Video Diffusion model. The improved Stable Video Diffusion model generates a video potential representation sequence through annealing sampling and cross-modal attention in the latent space diffusion process. Video denoising unit, used to transform the video potential representation sequence Each frame image is divided into local blocks and decomposed into amplitude spectrum M and phase spectrum P through FFT transform. The amplitude spectrum is adaptively smoothed and optimized based on the cross-frame consistency constraint and combined with the phase spectrum P. The inverse FFT is used to restore the time domain image block. The complete video frame is reconstructed by the weighted overlap method to output the final video sequence X′ final ; The video denoising unit specifically includes the following steps: S51, multi-scale control signal With the original image Input to the improved model based on the Stable Video Diffusion framework, first compressing the input image I into a potential representation through the 3D-VAE encoder At the same time, the multi-scale control signal S is mapped into a conditional embedding through the spatiotemporal convolutional network. In the diffusion denoising process, the latent variable z t Through the iterative update of the improved UNet architecture, its time-dependent residual block calculation form is: ò θ (z t ,t,c)=Conv 3D (AdaGN(z t ,t)+CrossAttn(z t ,c)), Among them, AdaGN(z t ,t) is the embedding vector injected at time step t, representing the adaptive group normalization layer; CrossAttn(z t ,c) is the cross-modal attention module, used to calculate z t c interaction weight And weighted fusion as conditional features; h = H / 8; w = W / 8; d = 4; t∈[1,T]; θ (z t ,t,c) represents the output of the denoising network; Conv 3D is a 3D convolutional layer that jointly models motion continuity in both spatial and temporal dimensions; H is the image height and W is the image width; S52, introduce annealing sampling strategy: in the denoising step t∈[T c ,T end ], and in t∈[1,T c ] gradually attenuates the weight of the multi-scale control signal Sγ(t)=min(1,(T c -t) / (T c -1)) to eliminate artifacts caused by over-constraints; finally, the potential representation is stabilized by the frequency domain stabilization module Post-processing outputs temporally smoothed video latent representation sequence is a learnable frequency domain filter; T end represents the termination time step of the annealing sampling strategy; T c Indicates the starting time step of the annealing sampling strategy.

Images