Camera track length-controllable video generation method and system based on video diffusion model

Abstract

The invention discloses a method and a system for generating a long video with a controllable camera track based on a video diffusion model. Comprising a camera track and initial frame preparation stage, a point cloud construction and multi-view image generation stage, a scale factor alignment optimization stage, a camera motion prior injection and noise initialization stage, a diffusion inversion generation stage and a sliding window time consistency fusion stage. Three-dimensional camera track modeling, projection reconstruction and diffusion processes are combined, a video content generation process is explicitly guided to be aligned with a track path set by a user, and long video generation which is reasonable in structure, natural in vision and continuous in time is achieved.

Description

Technical Field

[0001] This invention belongs to the field of controllable video generation technology, specifically relating to a method and system for generating long videos with controllable camera trajectory based on a video diffusion model. Background Technology

[0002] In recent years, generative diffusion models have rapidly become a research hotspot in computer vision due to their superior performance in image and video synthesis. Especially in video generation tasks, diffusion models can generate high-quality dynamic images while maintaining content consistency, thus finding wide application in scenarios such as short video generation, video completion, and style transfer. Existing video diffusion models such as SVD, VideoCrafter2, and AnimateDiff have all made significant progress in terms of generation quality and temporal consistency.

[0003] Existing video generation methods primarily employ two technical approaches: training-based camera control and training-free inference paradigms. The former fine-tunes the video diffusion model by incorporating camera pose or trajectory information, achieving a degree of camera viewpoint control. However, these methods generally rely on large-scale datasets with precise camera parameter annotations, resulting in high training costs, limited application scenarios, and a tendency to exhibit cross-frame distortion and discontinuities when generating long temporal videos. Typical examples include MotionCtrl and CameraCtrl, both requiring retraining on specially constructed datasets, whose generalization ability and efficiency are insufficient to meet the demands of real-world applications for long video generation and complex viewpoint control.

[0004] In contrast, training-free methods attempt to directly utilize the potential capabilities of pre-trained models by constructing initial noise distributions or explicit input control conditions to guide camera motion trajectories. These methods do not rely on additional training and have higher adaptability and generalization. However, in long video generation, due to the lack of inter-frame temporal modeling and contextual smoothing mechanisms, problems such as viewpoint jumps, content drift, and structural distortion often occur, severely affecting the quality and realism of the generated video. For example, while CamTrol can achieve a certain degree of camera control, its performance in terms of continuity and stability is poor when extended to long videos.

[0005] Existing generative video diffusion models have significant bottlenecks in long video generation tasks, mainly in three aspects:

[0006] First, existing methods have limited control over camera motion. Most methods only support simple translation and rotation operations, and are not precise enough for controlling complex motions (such as combined motions). In the absence of large-scale datasets with accurate camera annotations, control methods trained using these methods are difficult to generalize to unseen scenarios.

[0007] Second, inconsistencies across frames and screen flickering are common problems in the generation of long videos. Existing models are usually direct extensions of short video models and fail to effectively introduce temporal modeling mechanisms, resulting in significant discontinuities in the generated video over long time, especially when the camera viewpoint changes rapidly.

[0008] Third, there is currently a lack of a framework that can achieve high-quality, controllable, long-term video generation without retraining the model. Although existing research has explored video generation methods with flexible training, most have failed to achieve a good balance between control accuracy and temporal consistency.

[0009] Therefore, there is an urgent need for a new video generation technology that can achieve precise control of complex camera motion without relying on retraining, and maintain high quality, continuity and stability when generating long video sequences. Summary of the Invention

[0010] To address the problems in the background art, this invention provides a method and system for generating long videos with controllable camera trajectory based on a video diffusion model.

[0011] This invention proposes a long video generation method based on controllable camera trajectory, employing a three-stage process design: camera motion modeling, noise injection and diffusion generation, and sliding window consistency optimization. Each stage coordinates and works in tandem to ensure that the generated video not only conforms to the user-specified camera trajectory but also possesses excellent temporal consistency and visual continuity.

[0012] The three-stage generation process of this invention integrates text-to-image synthesis, depth-aware perspective projection achieved through point cloud transformation, and time-stabilized video diffusion achieved through window-adaptive attention reweighting. Notably, the method of this invention achieves unprecedented minute-level controllable perspective generation without requiring camera parameterization datasets or model retraining. The proposed sliding window mechanism fundamentally solves the flickering problem by dynamically balancing local motion consistency and global content preservation.

[0013] The technical solution adopted in this invention is as follows:

[0014] A method for generating long videos with controllable camera trajectory based on a video diffusion model includes the following steps:

[0015] 1) Initial Image Acquisition: Generate an image based on the initial text prompts or accept text-image pairs provided by the user;

[0016] 2) Point cloud construction: The depth map of the image is obtained by using a monocular depth estimation network, and then the corresponding 3D point cloud representation is generated by combining the intrinsic and extrinsic parameter matrices;

[0017] 3) Based on the extrinsic parameter matrix of each frame in the camera's motion trajectory, perform perspective transformation on the point cloud to generate a projected image; and fill in the hole areas of the projected image through the image inpainting module to obtain a complete inpainted image;

[0018] 4) Perform depth estimation again for each frame of the restored image and compare it with the point cloud of the previous frame. Optimize the scale factor by minimizing the point cloud error to achieve point cloud enhancement.

[0019] 5) Repeat steps 3 and 4 for all frames to obtain the complete multi-view image sequence V0;

[0020] 6) Predict the noise term using a pre-trained diffusion model, and then calculate the latent representation for the t-th step using a denoising formula.

[0021] 7) Use a sliding window strategy to smooth the latent representation at step t. Obtain the smoothed latent representation If the current step t is less than the number of sampling steps (e.g., 50), then return to step 6 to perform the next sampling step; otherwise, the final generated long video with controllable and smooth perspective is obtained.

[0022] Step 1) specifically refers to:

[0023] First, given an initial text prompt related to the final generated video, a pre-trained text-to-image diffusion model is used to generate corresponding images from the initial text prompt. Alternatively, it can accept text-image pairs provided by the user, with this initial image serving as the initial frame for the entire video sequence.

[0024] Step 2) specifically refers to:

[0025] A monocular depth estimation network is used to obtain a depth estimate D0∈R of the initial image. H×W ;

[0026] Point cloud P0 is obtained by upscaling pixels [I0,D0] to 3D space, as shown in the following formula:

[0027] P0 = φ([I0, D0], K, E0)

[0028] in, This represents the intrinsic parameter matrix of the camera. This represents the initial extrinsic parameter matrix, where Φ is the mapping function from RGBD to 3D point cloud.

[0029] Step 3) specifically refers to:

[0030] 3.1) Model the custom camera motion trajectory as a series of extrinsic parameter matrices. Each E i Contains a rotation matrix Translation vector E i =[R i |t i ];

[0031] The projected image at a specified position is obtained based on the extrinsic parameter matrix. The projected image at position i is obtained using the following formula:

[0032]

[0033] in, Represents the perspective projection function; P i Let i be the point cloud at position i;

[0034] The camera motion is user-defined and includes zoom, tilt, translation, rotation, and combinations thereof;

[0035] 3.2) Image restoration model adopted The formula for optimizing the projected image is as follows:

[0036]

[0037] Among them, the binary mask M i Used to mark the area with holes.

[0038] Step 4) specifically involves:

[0039] 4.1) The repair results Depth estimates are generated using a monocular depth estimator. Optimize the depth scaling factor d of the monocular depth estimator i The optimized formula is as follows:

[0040]

[0041] Where φ(·) represents 3D back projection; d is a scaling factor used to scale the relative depth map. With P i-1 Alignment of absolute coordinates within;

[0042] 4.2) Through Repair results Reproject back into 3D space to enhance the point cloud;

[0043] Step 6) specifically involves:

[0044] 6.1) Constructing a noise prior using forward diffusion:

[0045]

[0046] in, It serves as the starting point for the denoising process of the video diffusion model and is also the noise prior for forward diffusion denoising in the video diffusion model. α t t0 represents the variance of the video diffusion model scheduler, and t0 represents the number of forward noise addition steps of the video diffusion model, which is also the starting step number of the reverse noise reduction of the video diffusion model.

[0047] 6.2) The reverse learning process of the video diffusion model is achieved through a noise prediction network ∈ θ from Iterative denoising to This network operates on time series data:

[0048]

[0049] Where, α t =1-β t ,σ t To control for randomness, t belongs to 1 to t0 and ∈ ~N(0,I); the learnable parameter θ is optimized to predict noise components through the following objective:

[0050]

[0051] Among them, c t This represents the text embedding vector obtained by encoding the initial text using a text encoder.

[0052] Step 7) specifically involves:

[0053] Given a window size w and a stride s, for each time step t in a long video, what is the smoothed latent representation? The calculation is as follows:

[0054]

[0055] in, The latent representation after smoothing by the sliding window represents the latent video representation at step t-1 of the video diffusion model denoising process; N represents the total number of video frames; n represents the frame index, with values ​​n = 1, 2, ..., N; [i k ,j k [k*s, k*s+w] represents the frame interval covered by the k-th sliding window; K represents the number of the k-th sliding window. n This indicates the number of windows covered in the nth frame.

[0056] The complete algorithm is described in detail below:

[0057] (1) Camera trajectory and initial frame preparation stage: The system first receives the user-input text description (Prompt) or initial image, and loads the user-specified or predefined camera motion trajectory sequence. If no initial image is provided, an open-source Text-to-Image model (such as SDXL or DALLE) is used to generate an image based on the text prompt. This image will serve as the initial frame for the entire video sequence.

[0058] (2) Point Cloud Construction and Multi-View Image Generation Stage: The initial image is processed using a monocular depth estimation model to obtain a depth map. A 3D point cloud representation is generated by combining intrinsic and extrinsic parameters. Subsequently, the system performs perspective transformation on the point cloud based on the extrinsic parameters of each frame in the camera trajectory, generating a projected image. Hollow areas are then filled using an image inpainting module to obtain a complete reference image. This process is repeated for all frames to obtain a complete multi-view image sequence.

[0059] (3) Scale factor alignment optimization stage: In order to improve the structural consistency between multiple frames, the depth of each frame repaired image is estimated again, and compared with the point cloud of the previous frame. The scale factor is optimized by minimizing the point cloud error, so that the point cloud of the current frame is aligned with the structure of the previous frame, thereby enhancing spatial continuity.

[0060] (4) Camera motion prior injection and noise initialization stage: Based on the multi-frame sequence, construct the diffusion initial latent variable corresponding to the time step, and implicitly inject the camera motion information into the starting point of the diffusion process.

[0061] (5) Diffusion Inversion Generation Stage: In the diffusion modeling stage, the diffusion model parameters are not trained; only their denoising capabilities are used. In each inversion step, the system predicts the noise term using the pre-trained diffusion model and generates the latent variables for the next time step according to the following update strategy.

[0062] (6) Sliding window temporal consistency fusion stage: In order to improve the temporal stability and continuity in long-sequence videos, the system introduces a sliding window strategy. For the latent variable representation of each frame, multiple windows are used for average fusion to prevent inter-frame jumps and ensure content consistency.

[0063] Beneficial effects of the present invention

[0064] (1) This invention proposes a training-free method framework based on camera trajectory control for long video generation. It enables precise control of viewpoint movement solely through point cloud reconstruction and noise prior injection, without requiring any weight updates to the pre-trained video diffusion model. This invention combines 3D camera trajectory modeling, projection reconstruction, and diffusion processes, explicitly guiding the video content generation process to align with the user-defined trajectory path, achieving the generation of long videos with reasonable structure, natural visuals, and temporal continuity. On multiple public benchmarks (such as VBench), this invention achieves optimal performance in key indicators such as subjective consistency, background consistency, trajectory reconstruction error, and temporal flicker. For example, in the camera trajectory evaluation metric, it achieves an ATE error of 0.27, and in the RPE-R and RPE-T metrics, it achieves 1.81 and 0.13 respectively, both superior to methods such as MotionCtrl and CamTrol, fully validating the effectiveness and leading-edge nature of this invention.

[0065] (2) The method of the present invention exhibits significant advantages in temporal consistency and visual stability in long video generation tasks. Experiments show that after adopting the sliding window temporal fusion mechanism proposed in this invention, the generated sequence is significantly better than the traditional frame-by-frame diffusion method in terms of object structure, background texture and viewpoint coherence, effectively alleviating the common cross-frame flickering problem in video sequences, and making the video generation more realistic and coherent in subjective experience. Attached Figure Description

[0066] Figure 1 This is a diagram illustrating a long video generation method based on controllable camera trajectory.

[0067] Figure 2 This is a demonstration image of the controllable long video generation of the present invention. Specific Implementation

[0068] The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples.

[0069] 1. Camera motion modeling stage

[0070] To ensure camera controllability in the generated video content, this invention injects camera motion signals into the initial latent noise. First, an explicit camera motion representation is established. Unlike existing methods (such as CamCtrl or MotionCtrl) that encode camera trajectories into a diffused U-Net via an encoder network, which requires extensive labeled data for fine-tuning and carries the risk of overfitting, and unlike Direct-a-video which constructs camera motion through simple image operations but cannot handle complex trajectories such as rotation, this invention, inspired by CamTrol, proposes a point cloud-based camera motion modeling method comprising three key components.

[0071] Point cloud construction: First, given an initial text cue related to the final generated video, a pre-trained text-to-image diffusion model is used to generate corresponding images from the initial text cue. Alternatively, it can accept text-image pairs provided by the user. Then, a monocular depth estimation network is used to obtain the depth estimate D0∈R of the image. H×W .

[0072] Point cloud P0 is obtained by upscaling pixels [I0,D0] to 3D space, as shown in the following formula:

[0073] P0 = φ([I0, D0], K, E0)

[0074] in, This represents the intrinsic parameter matrix of the camera, while This represents the initial extrinsic parameter matrix. Φ is the mapping function from RGBD to 3D point cloud.

[0075] Trajectory-guided rendering: This invention models camera motion as a series of extrinsic parameter matrices. Each E i =[R i |t i Contains a rotation matrix Translation vector

[0076] The projected image at a specified position is obtained based on the extrinsic parameter matrix. The projected image at position i is obtained using the following formula:

[0077]

[0078] in, This represents the perspective projection function. P i Let i be the point cloud at position i.

[0079] This makes it possible to accurately simulate complex camera trajectories, including zoom, tilt, translation, rotation, and combinations thereof.

[0080] Consistent Multi-View Synthesis: Direct projection of sparse point clouds introduces artifacts and holes, thus reducing temporal consistency. This invention employs an image inpainting model. To optimize projection:

[0081]

[0082] Among them, the binary mask M i Mark the area with holes.

[0083] Repair results pass Reproject back into 3D space to enhance the point cloud.

[0084] Repair results Depth estimates are generated using a monocular depth estimator. Because monocular depth estimators only recover relative scale Inter-frame inconsistency depth coefficient d i This can lead to 3D positional drift. Optimize the depth scaling factor d of the monocular depth estimator. i The specific optimization formula is as follows:

[0085]

[0086] Where φ(·) represents 3D back projection, M represents the overlapping region between continuous point clouds, and d is used as a scaling factor to adjust the relative depth map. With P i-1 Alignment with absolute coordinates.

[0087] 2. Noise Injection and Diffusion Generation Stage

[0088] Image sequences synthesized from the camera motion modeling stage Predefined camera motion is captured, but may contain artifacts and lack temporal dynamics. The key to this invention is to construct an initial noise that can preserve motion patterns while removing artifacts.

[0089] Motion Alignment Noise Construction: This invention recognizes that initial noise significantly controls the layout of generated content. Based on the finding that camera motion manifests as inter-pixel displacement, this invention uses forward diffusion to construct a noise prior:

[0090]

[0091] in, α t t0 represents the variance of the diffusion model scheduler, and t0 represents the number of forward noise addition steps of the video diffusion model, which is also the starting step number of the reverse noise reduction of the video diffusion model. It serves as the starting point for the denoising process of the video diffusion model and is also a noise prior for the forward diffusion of the video diffusion model.

[0092] This formula preserves the relative pixel offset of the encoded camera motion in V0. As analyzed, the injected Gaussian noise inherently fills in the missing regions through the denoising process, thus jointly addressing the artifact removal and motion preservation issues.

[0093] The reverse learning process of the video diffusion model is achieved through a noise prediction network. θ from Iterative denoising to This network operates on time series data:

[0094]

[0095] Where αt = 1 - βt, σt controls randomness, t belongs to 1 to t0, and ∈ ~N(0,I). The learnable parameter θ is optimized to predict the noise component through the following objective:

[0096]

[0097] This formula models the latent representation of the video simultaneously. The time consistency is ensured, and the corresponding conditional signals are integrated through cross-attention mechanisms (such as text prompts). t This expands upon standard diffusion training.

[0098] 3. Sliding window consistency optimization stage

[0099] While the camera motion modeling and noise injection described above can generate videos with a controllable viewpoint, this invention observes undesirable frame flickering and temporal instability when extending the duration of the generated videos. To address this issue, this invention employs a sliding window strategy to smooth the underlying temporal representation.

[0100] Specifically,

[0101] Given a window size w and a step size s, this invention aggregates the latent outputs within each time window and calculates the final output by averaging the overlapping regions. Formally, for each time step t in a long video, the smoothed latent representation... The calculation is as follows:

[0102]

[0103] in, The latent representation after smoothing by the sliding window represents the latent representation of the video at step t-1 in the video diffusion model denoising process; N represents the total number of video frames; n represents the frame index, with values ​​n = 1, 2, ..., N; [i k j k [k*s, k*s+w] represents the frame interval covered by the k-th sliding window; k represents the number of the k-th sliding window; K n Represents the number of windows covered in the nth frame, used for normalized averaging.

[0104] This averaging process effectively reduces high-frequency time noise and produces more stable video output.

[0105] In a specific implementation, taking the generation of a 64-frame video with a resolution of 320×512 as an example, the method for generating long videos with controllable camera trajectory based on a video diffusion model is described in detail. Figure 1 This is a flowchart illustrating the method framework of the present invention, showing the entire process from text input to final video generation; Figure 2 The following diagrams illustrate the effects of this invention, showing examples of keyframes in the generated video; Table 1 compares the performance of the method of this invention with other methods; Table 2 compares the ablation effects of each module of the method of this invention.

[0106] Demonstration of generated results: The video generated by this invention strictly follows the user-specified camera trajectory (such as a combination of zoom, translation, and rotation) and maintains temporal consistency. For example... Figure 2 As shown, with the prompt "a troop of monkey on top of a mountain + zoom out and tilt up", the perspective transitions smoothly in frames 1, 16, 32, 48, and 64, without content drift or structural distortion.

[0107] Table 1

[0108]

[0109] Performance Comparison Analysis: In tests on 500 cue-track pairs (VBench benchmark), this invention outperformed the baseline in both camera control accuracy and video quality (Table 1). Key metrics include: Camera trajectory error: Absolute trajectory error (ATE) was 0.27, rotation error (RPE-R) was 1.81, and translation error (RPE-T) was 0.13, all lower than the training method MotionCtrl (ATE 0.41) and the training-free method CamTrol (ATE 0.47). Video quality metrics: Subject consistency (Sub) reached 0.95, background consistency (Back) reached 0.97, and motion smoothness (Motion) reached 0.98, superior to all baselines.

[0110] Component Ablation Study: To verify the necessity of the core components (camera motion prior injection and sliding window), the following configurations were compared on the same test set: No Camera Prior (CamMo Prior): Using only the sliding window improves video quality (Sub 0.95, Back 0.96), but the viewpoint cannot be controlled. No Sliding Window (SW): Injecting only the camera prior results in a high ATE error (0.47) and severe inter-frame flicker. Complete Method (CamMo Prior + SW): Using both methods together reduces the ATE to 0.27, achieving optimal video consistency.

[0111] This embodiment demonstrates that by explicitly modeling camera trajectory using point clouds, injecting viewpoint displacement with noise priors, and balancing local motion with global content using a sliding window, this invention achieves precise viewpoint control (such as scaling and rotational composite motion) for 64-frame long videos without training, and its temporal consistency is superior to existing methods (RPE-R error reduced by more than 40%). Ablation experiments show that the sliding window mechanism is key to solving inter-frame flicker (flicker index Fliker improved to 0.98), while camera motion priors ensure trajectory accuracy (ATE 0.27). This method is applicable to scenarios such as UAV aerial photography and virtual reality, expanding the practical boundaries of training free video generation.

[0112] Table 2

[0113]

Claims

1. A method for generating long videos with controllable camera trajectory based on a video diffusion model, characterized in that, Includes the following steps: 1) Initial Image Acquisition: Generate an image based on the initial text prompts or accept text-image pairs provided by the user; 2) Point cloud construction: The depth map of the image is obtained by using a monocular depth estimation network, and then the corresponding 3D point cloud representation is generated by combining the intrinsic and extrinsic parameter matrices; 3) Based on the extrinsic parameter matrix of each frame in the camera's motion trajectory, perform perspective transformation on the point cloud to generate a projected image; The image restoration module fills in the voids in the projected image to obtain a fully restored image. 4) Perform depth estimation again for each frame of the restored image and compare it with the point cloud of the previous frame. Optimize the scale factor by minimizing the point cloud error to achieve point cloud enhancement. 5) Repeat steps 3 and 4 for all frames to obtain the complete multi-view image sequence V0; 6) Predict the noise term using a pre-trained diffusion model, and then calculate the latent representation for the t-th step using a denoising formula. 7) Use a sliding window strategy to smooth the latent representation at step t. Obtain the smoothed latent representation If the current step t is less than the number of sampling steps, return to step 6 to perform the next sampling step; otherwise, a long video with a controllable and smooth viewing angle will be generated.

2. The method for generating long videos with controllable camera trajectory based on a video diffusion model according to claim 1, characterized in that, Step 1) specifically refers to: First, given an initial text prompt, a pre-trained text-to-image diffusion model is used to generate corresponding images from the initial text prompt. Alternatively, it can accept text-image pairs provided by the user, with this initial image serving as the initial frame for the entire video sequence.

3. The method for generating long videos with controllable camera trajectory based on a video diffusion model according to claim 1, characterized in that, Step 2) specifically refers to: A monocular depth estimation network is used to obtain a depth estimate D0∈R of the initial image. H×W ; Point cloud P0 is obtained by upscaling pixels [I0,D0] to 3D space, as shown in the following formula: P0 = φ([I0,D0],K,E0) in, This represents the intrinsic parameter matrix of the camera. This represents the initial extrinsic parameter matrix, where Φ is the mapping function from RGBD to 3D point cloud.

4. The method for generating long videos with controllable camera trajectory based on a video diffusion model according to claim 1, characterized in that, Step 3) specifically refers to: 3.1) Model the custom camera motion trajectory as a series of extrinsic parameter matrices. Each E i Contains a rotation matrix Translation vector The projected image at a specified position is obtained based on the extrinsic parameter matrix. The projected image at position i is obtained using the following formula: in, Represents the perspective projection function; P i Let i be the point cloud at position i; The camera motion is user-defined and includes zoom, tilt, translation, rotation, and combinations thereof; 3.2) Image restoration model adopted The formula for optimizing the projected image is as follows: Among them, the binary mask M i Used to mark areas with holes.

5. The method for generating long videos with controllable camera trajectory based on a video diffusion model according to claim 1, characterized in that, Step 4) specifically involves: 4.1) The repair results Depth estimates are generated using a monocular depth estimator. Optimize the depth scaling factor d of the monocular depth estimator i The optimized formula is as follows: Where φ(·) represents 3D back projection; d is a scaling factor used to scale the relative depth map. With P i-1 Alignment of absolute coordinates within; 4.2) Through Repair results Reproject back into 3D space to enhance the point cloud.

6. The method for generating long videos with controllable camera trajectory based on a video diffusion model according to claim 1, characterized in that, Step 6) specifically refers to: 6.1) Constructing a noise prior using forward diffusion: in, It serves as the starting point for the denoising process of the video diffusion model and is also the noise prior for forward diffusion denoising in the video diffusion model. α t t0 represents the variance of the video diffusion model scheduler, and t0 represents the number of forward noise addition steps of the video diffusion model, which is also the starting step number of the reverse noise reduction of the video diffusion model. 6.2) The reverse learning process of the video diffusion model is achieved through a noise prediction network ∈ θ from Iterative denoising to This network operates on time series data: Where, α t =1-β t ,σ t To control for randomness, t belongs to 1 to t0 and ∈ ~N(0,I); the learnable parameter θ is optimized to predict noise components through the following objective: Among them, c t This represents the text embedding vector obtained by encoding the initial text using a text encoder.

7. The method for generating long videos with controllable camera trajectory based on a video diffusion model according to claim 1, characterized in that, Step 7) specifically involves: Given a window size w and a stride s, for each time step t in a long video, what is the smoothed latent representation? The calculation is as follows: in, The latent representation after smoothing by the sliding window represents the latent video representation at step t-1 of the video diffusion model denoising process; N represents the total number of video frames; n represents the frame index, with values ​​n = 1, 2, ..., N; [i k ,j k [k*s, k*s+w] represents the frame interval covered by the k-th sliding window; K represents the number of the k-th sliding window. n This indicates the number of windows covered in the nth frame.

8. A terminal, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to, when executing the computer program, implement the camera trajectory controllable long video generation method based on the video diffusion model as described in claim 1.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the camera trajectory controllable long video generation method based on a video diffusion model as described in claim 1.

Images