Digital human video generation method with camera control
By combining the Plücker camera encoder and diffusion converter, the problem of insufficient camera control in digital human video generation is solved, achieving high-quality camera motion synchronization and long video generation, which is suitable for film and television production.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- GIANT MOBILE TECH CO LTD
- Filing Date
- 2026-03-09
- Publication Date
- 2026-06-05
AI Technical Summary
In existing technologies, digital human video generation models lack camera control capabilities, resulting in fixed video backgrounds and perspectives, an inability to produce realistic parallax effects, poor text control precision, difficulty in multimodal fusion, and high training costs.
By employing a Plücker camera encoder, a camera control injection mechanism based on a diffusion transformer, and a post-adaptor module, combined with full-parameter training and LoRA fine-tuning strategies, a time-step-aware dynamic window range strategy is designed to generate videos that conform to the camera trajectory.
It achieves precise camera control, high-quality lip-sync and motion synchronization, supports long video generation, has high training efficiency, and is suitable for film and television production.
Smart Images
Figure CN122156413A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and computer vision technology, and in particular to a method for generating digital human videos with camera control. Background Technology
[0002] Current mainstream speaker generation methods typically suffer from the following shortcomings:
[0003] 1. Lack of camera control: Most models generate videos with fixed backgrounds and camera angles, or simply simulate camera movement by post-processing the generated 2D video by panning or scaling. This method cannot produce realistic parallax effects, resulting in a lack of depth in the image.
[0004] 2. Limitations of Text Control: Some existing video generation models attempt to control camera movement through text prompts (such as "zoom out" or "pan left"), but this approach has significant drawbacks:
[0005] Low compliance: The model often ignores camera movement instructions, resulting in static or incorrectly moving images.
[0006] Poor control precision: The text cannot accurately describe the amplitude of camera movement, the time points of change (such as when to start circling after zooming in), speed changes, and complex irregular trajectories, which cannot meet the needs of professional content production.
[0007] 3. Difficulty in multimodal fusion: Simultaneously introducing audio-driven and camera-controlled features can easily lead to feature conflicts. For example, large camera movements may interfere with facial features, resulting in face ID distortion and decreased lip-sync accuracy.
[0008] 4. High training cost: In order to introduce new control conditions (such as camera parameters), it is often necessary to train a huge video generation model from scratch, which consumes huge computing resources.
[0009] Therefore, it is necessary to provide a digital human video generation method with camera control that can simultaneously generate a human speaking video with specific camera movement effects based on reference images, driving audio, text prompts, and camera trajectory files. Summary of the Invention
[0010] The purpose of this invention is to provide a method for generating digital human videos with camera control, which can simultaneously generate a video of a person speaking with specific camera movement effects based on a reference image, driving audio, text prompts, and camera trajectory files.
[0011] To address the problems existing in the prior art, this invention provides a method for generating digital human videos with camera control, comprising the following steps:
[0012] S1: Inject camera control into the model architecture design to generate a deep architecture from physical parameters to feature injection. The deep architecture includes the Plücker camera encoder, a diffusion transform-based camera control injection mechanism, and a post-adaptor module.
[0013] S2: Set up a hybrid fine-tuning strategy, using a pre-trained digital human video generation model as the base model, and adopt a hybrid training strategy of full parameter training and LoRA fine-tuning.
[0014] S3: Sets a time-step-aware dynamic window range strategy, which adopts a two-layer loop mechanism: the outer loop is the reverse denoising process of the diffusion model, and the inner loop is the sliding window processing of the entire video sequence at the current time step, which is used to support the generation of long videos with a duration exceeding the single inference window limit.
[0015] Optionally, in the camera-controlled digital human video generation method, the diffusion transformer is called DiT.
[0016] Optionally, in the digital human video generation method with camera control, the Plücker camera encoder includes the following functions:
[0017] Input and Function: Used to receive the camera extrinsic and intrinsic parameter matrices for each frame and convert them into feature tensors rich in geometric information and aligned with the video latent space;
[0018] Gaze embedding generation: Calculate the gaze origin and gaze direction for each pixel in each frame of the image based on the camera parameters, and generate the Plücker embedding.
[0019] Temporal dimension compression: In order to adapt to the temporal compression characteristics of the VAE latent space in the DiT model, this encoder adopts a lossless transformation strategy based on repetition and reshaping.
[0020] Optionally, in the digital human video generation method with camera control, the algorithm of the Plücker camera encoder is as follows:
[0021] The first frame of the input Plücker embed is copied 4 times, changing its timing length from 81 to 84.
[0022] By using tensor reshaping and transpose operations, the data from four adjacent time frames are stitched together along the channel dimension.
[0023] The final output is the camera feature tensor.
[0024] Optionally, in the digital human video generation method with camera control, the camera control injection mechanism based on the diffusion transformer deeply integrates the camera control signal into each DiT module to achieve block-by-block injection, thereby providing the model with compliance with camera control.
[0025] The working principle of the camera control injection mechanism based on the diffusion converter is as follows:
[0026] Each DiT module contains an independent camera embedded in a projector;
[0027] The camera-embedded projector receives a 24-channel feature tensor from the Plücker camera encoder.
[0028] The width and height of the tensor are downsampled to the same size as the noise latent variable using the PixelUnshuffle operator.
[0029] Increase its dimension to the same number as the noise latent variable by using 2D convolution.
[0030] Deep semantic features are extracted using residual blocks.
[0031] Optionally, in the camera-controlled digital human video generation method, the post-adaptor module refines and adjusts the features processed by the camera control injection mechanism based on the diffusion transformer, enhancing the model's ability to coordinate digital human lip alignment and camera motion.
[0032] Optionally, in the digital human video generation method with camera control, full-parameter training is used to train the Camera Embedding Projector and Adapter within each DiT block with full parameters, so that they can quickly adapt to the camera control task;
[0033] LoRA fine-tuning is used to fine-tune the original Self-Attention layer using LoRA technology, preserving the original model's high-quality generation capabilities while adapting to new multimodal inputs.
[0034] Optionally, in the camera-controlled digital human video generation method, the time-step-aware dynamic window range strategy is set as follows:
[0035] Dynamic window offset: At each denoising time step, the starting position of the sliding window will be dynamically offset according to the preset step size, so that each frame is in a different relative position within the window at different denoising stages, thereby expanding the temporal context receptive field of each frame and eliminating the seam effect caused by fixed window division.
[0036] Non-wrap and overlap mechanism: Dynamic offset is introduced while maintaining window overlap;
[0037] Boundary adaptive constraints:
[0038] Head protection: When the accumulated offset causes the effective length of the first segment to be less than a set threshold, the offset is automatically reset to prevent the window from being too short and affecting the generation quality;
[0039] End-of-video padding: If the length of a segment at the end of a video is less than the minimum window requirement, the strategy will automatically extend its starting position forward to meet the minimum length constraint. This also prevents the window from being too short and affecting the generation quality.
[0040] Compared with the prior art, the present invention has the following advantages:
[0041] (1) Precise and physically consistent camera control: Thanks to the lossless encoding of gaze information by the Plücker camera encoder and the depth injection mechanism inside DiT, this invention can accurately follow complex 3D camera trajectories. Compared with text control or simple 2D transformation, the video generated by this method solves the problems of "unruly camera movement" or "lack of perspective".
[0042] (2) High-quality lip movements and actions: Based on the fine-tuning of the pre-trained digital human model and combined with the design of the rear adapter, it is ensured that the lip movements of the characters are still accurately aligned and the facial structure remains stable even under large camera movements.
[0043] (3) Training is efficient and flexible: The hybrid training strategy of using LoRA of existing modules combined with retraining of new modules saves memory and training time compared with full fine-tuning, and can flexibly adapt to different base models.
[0044] (4) Support for long videos: It solves the problem of continuity in the generation of long shots and is applicable to a wider range of film and television production scenarios.
[0045] (5) This invention proposes a digital human video generation method with camera control, which allows users to input predefined camera trajectory files. The model can generate dynamic videos that not only conform to the image of the reference person and the lip movements of the driving audio, but also strictly follow the motion of the input camera trajectory.
[0046] (6) This invention proposes a digital human video generation model based on the DiT (Diffusion Transformer) architecture. Based on the Wan2.1 base model, it innovatively designs a camera control injection model architecture and proposes a long video inference mechanism. Attached Figure Description
[0047] Figure 1 A flowchart of a digital human video generation method provided in an embodiment of the present invention. Detailed Implementation
[0048] The specific embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. The advantages and features of the present invention will become clearer from the following description. It should be noted that the drawings are all in a very simplified form and use non-precise proportions, and are only used to facilitate and clarify the illustration of the embodiments of the present invention.
[0049] In the following, if the methods described herein include a series of steps, the order of these steps presented herein is not necessarily the only order in which these steps can be performed, and some of the steps described may be omitted and / or some other steps not described herein may be added to the method.
[0050] To address the problems existing in the prior art, this invention provides a method for generating digital human videos with camera control, such as... Figure 1 As shown, it includes the following steps:
[0051] S1: Inject camera control into the model architecture design to generate a deep architecture from physical parameters to feature injection. The deep architecture includes the Plücker camera encoder, a camera control injection mechanism based on diffusion transformer, and a post-adaptor module. Diffusion transformer is abbreviated as DiT.
[0052] (1) The Plücker camera encoder includes the following functions:
[0053] Input and Function: This module receives the camera extrinsic matrix (rotation matrix R + translation matrix T) and intrinsic matrix (K) for each frame, and converts them into a feature tensor rich in geometric information and aligned with the video latent space.
[0054] Gaze embedding generation: First, based on the camera parameters, the gaze origin (Origin, o) and gaze direction (Direction, d) corresponding to each pixel in each frame are calculated, generating a Plücker embedding of shape [B, F, 6, H, W] (6 channels = 3 channels origin + 3 channels direction), where B represents the batch dimension of the Plücker embedding tensor, F represents the temporal frame dimension of the Plücker embedding tensor, 6 represents the channel dimension of the Plücker embedding tensor, and H and W represent the height and width dimensions of the Plücker embedding tensor, respectively.
[0055] Temporal compression: In order to adapt to the temporal compression characteristics (usually 4x compression) of the VAE (Variational Autoencoder) potential space in the DiT model, this encoder does not use traditional convolutional downsampling, but instead adopts a lossless transformation strategy based on repetition and reshape.
[0056] The algorithm for the Plücker camera encoder is as follows:
[0057] The first frame of the input Plücker embed is copied 4 times, changing its timing length from 81 to 84.
[0058] The data from four adjacent time frames are stitched together along the channel dimension by using tensor reshape and transpose operations.
[0059] The final output shape is [B, 24, F]. latent [, H, W] (B represents the sign of the batch dimension of the camera feature tensor, 24 represents the value of the channel dimension of the tensor, F latent The symbol F represents the temporal frame dimension of a tensor. latent The difference between F and F is F latent It exists in the latent space, so it is smaller than F. The relationship between them is F. latent = F / 4 + 1, where H and W are symbols representing the height and width dimensions of the Plück embedding tensor, respectively. The number of channels is expanded from 6 to 24 (6...). 4) The time dimension is compressed from F to F latent = F / 4 + 1. This design ensures that no high-frequency camera motion information is lost during time downsampling.
[0060] (2) Deep Camera Injection Mechanism Based on DiT
[0061] This invention abandons the traditional approach of simply superimposing control signals on the DiT input layer, and instead deeply integrates camera control signals into each DiT module, achieving block-by-block injection, thereby providing the model with greater compliance with camera control.
[0062] Camera Embedding Projector: Within each DiT module, there is an independent camera embedding projector. It receives a 24-channel feature tensor from the Plücker camera encoder, first downsamples the width and height of the tensor to the same size as the noisy latent using the PixelUnshuffle operator, then increases its dimension to the same dimension as the noisy latent using 2D convolution, and finally extracts deep semantic features using some residual blocks.
[0063] Injection path: After the first Layer Norm layer of the DiT module, the camera features output by the Camera EmbeddingProjector are directly added to the backbone features (i.e., latent noise variables), and the fused features are then input into the Self-Attention module. This design ensures that camera motion information can directly guide the spatial modeling of the self-attention mechanism, thereby generating dynamic images that conform to perspective.
[0064] (3) Rear adapter module (Adapter)
[0065] A trainable Adapter module is added after the output of the self-attention layer of each DiT module.
[0066] This module is used to further refine and adjust the features after self-attention processing, enhance the model's ability to coordinate digital lip alignment and camera motion, and alleviate multimodal signal conflicts.
[0067] S2: Set up a hybrid fine-tuning strategy, using a pre-trained digital human video generation model as the base model, and adopt a hybrid training strategy of full parameter training and LoRA fine-tuning.
[0068] Specifically, full-parameter training: Full-parameter training is performed on the newly added modules: the Camera EmbeddingProjector and Adapter within each DiT block, enabling them to quickly adapt to camera control tasks. The Plücker CameraEncoder requires no training.
[0069] LoRA Fine-tuning: The original Self-Attention layer is fine-tuned using LoRA (Low-Rank Adaptation) technology, preserving the original model's high-quality generation capabilities while adapting to new multimodal inputs. The parameters of other DiT modules remain frozen.
[0070] S3: Sets a timestep-aware dynamic window range strategy, which employs a two-layer loop mechanism: the outer loop is the reverse denoising process of the diffusion model, and the inner loop is the sliding window processing of the entire video sequence at the current time step, used to support the generation of long videos whose duration exceeds the single inference window limit.
[0071] The strategy for setting the dynamic window range for time-step awareness is as follows:
[0072] Dynamic Window Shift: At each denoising timestep, the starting position of the sliding window is dynamically shifted according to a preset step size. This allows each frame to be in a different relative position within the window at different denoising stages, thereby significantly expanding the temporal contextual receptive field of each frame and effectively eliminating the "seam" effect caused by fixed window division.
[0073] Non-rolling and overlapping mechanism: Unlike some existing methods that use a rolling strategy to connect the beginning and end of the video, this invention does not use rolling to avoid interference caused by the visual discontinuity of the beginning and end frames; at the same time, unlike some existing methods that use fixed position overlap, this invention introduces the above-mentioned dynamic offset while maintaining window overlap.
[0074] Boundary adaptive constraints:
[0075] Head protection: When the accumulated offset causes the effective length of the first segment to be less than the set threshold, the offset is automatically reset to prevent the window from being too short and affecting the generation quality.
[0076] End-of-video padding: If the length of a segment at the end of a video is less than the minimum window requirement, the strategy will automatically extend its starting position forward to meet the minimum length constraint. This is also to prevent the window from being too short and affecting the generation quality.
[0077] In one embodiment,
[0078] I. Model Architecture Configuration:
[0079] Base model: The Wan2.1 Video Transformer model is used, pre-trained on a large-scale digital human video dataset.
[0080] Plücker Camera Encoder Implementation Details:
[0081] Line of sight calculation: For each pixel coordinate (u, v), calculate the line of sight direction d and origin o based on the intrinsic parameter K and the extrinsic parameters R, T.
[0082] Tensor Transform: The input tensor shape is (B, F, H, W, 6). First, the channel dimension is pre-formed to obtain (B, 6, F, H, W). Then, in the time dimension, the first frame is copied 4 times, and then the 4 consecutive frames are reassembled into the channel dimension using `reshape` and `transpose` operations, resulting in an output shape of [B, 24, F]. latent [, H, W], where H and W are the original width and height of the reference image.
[0083] DiT module improvements:
[0084] For each DiT Block in the model (N in total):
[0085] 1. Layer Norm 1: Normalizes the input Latent X.
[0086] 2. Camera Injection: Calculate X fused =LayerNorm(X)+cameraembeddingprojector(C emb CameraEmbeddingProjector is a lightweight convolutional network.
[0087] 3. Self-Attention: Calculate X attn = SelfAttention(X fused LoRA weights are injected here.
[0088] 4. Adapter: Calculates X adapted =Adapter(X attn The Adapter is a lightweight, fully connected layer.
[0089] 5. Subsequent layers: Layer Norm -> Cross-Attention (Text-Image & Audio-Image) -> FFN.
[0090] II. Implementation of the Training Process
[0091] Data preparation:
[0092] Video data: Hundreds of hours of high-resolution digital human speaking video.
[0093] Camera trajectory extraction: The camera pose trajectory of each video segment is extracted using the VGGT (Visual Geometry Ground Truth) model.
[0094] Plücker Embedding generation formula:
[0095] For any pixel point p = (u, v, 1) on the image plane T The corresponding line of sight d in the world coordinate system world and the origin o world The calculation is as follows:
[0096] ;
[0097] d camera = K -1 p;
[0098] d world = R d camera ;
[0099] o world = T;
[0100] Plücker Embedding(u, v) = o world ⊕d world ,
[0101] Where ⊕ represents the concatenation of channel dimensions, the resulting embedded graph contains 6 channels.
[0102] Loss function: Flow Matching Loss is used to calculate the difference between the predicted velocity field and the true velocity field.
[0103] Optimization strategy:
[0104] Freeze most of the parameters of the DiT trunk.
[0105] Activate the gradients of the Camera Embedding Projector, Adapter, and Self-Attention LoRA.
[0106] Set a differentiated learning rate: Use a higher learning rate (e.g., 5) for new modules (Projector / Adapter). 10 -5 The LoRA part uses a lower learning rate (e.g., 1). 10 -5 ).
[0107] III. Implementation of Long-Video Reasoning
[0108] Parameter definition:
[0109] f: Window Length for a single inference operation.
[0110] o: Window overlap length.
[0111] p: Shift Step.
[0112] m: Maximum offset threshold.
[0113] n: Minimum Clip Length.
[0114] α: Current offset (Shift Offset), initialized to 0.
[0115] z is the video latent variable, initially set as noise. .
[0116] Detailed execution steps:
[0117] 1. Initialization: Set the total video length noise latent variable z T Initialize offset α=0.
[0118] 2. Outer loop (denoising): For time steps t=T,T-1,…,1:
[0119] Offset reset: If α > m, then reset α = 0.
[0120] First window calculation:
[0121] Calculate the starting point s = -α.
[0122] Calculate the termination point e = s + f.
[0123] Boundary correction s = max(0, s).
[0124] Inner loop (sliding window): when e < At that time, execute:
[0125] Model prediction: .
[0126] Window update:
[0127] If e < (Condition check):
[0128] s=eo (sliding step).
[0129] Endpoint correction: If s + f < , then e = s + f; otherwise e = (handling the end).
[0130] Minimum length constraint: If e - s < n, then s = e - n (to ensure the length of the end segment).
[0131] Offset accumulation: α new = α old + p.
[0132] 3. End: After the loop is completed, the final video z0 is obtained.
[0133] Compared with the prior art, the present invention has the following advantages:
[0134] (1) Precise and physically consistent camera control: Benefiting from the lossless encoding of line-of-sight information by the Plücker camera encoder and the depth injection mechanism inside DiT, the present invention can accurately follow complex 3D camera trajectories. Compared with text control or simple 2D transformations, the videos generated by this method solve the problems of "unruly camera movement" or "lack of perspective".
[0135] (2) High-quality lip sync and movements: Based on fine-tuning of the pre-trained digital human model and combined with the design of the post Adapter, it ensures that under large camera movements, the lip sync of the character is still accurately aligned and the facial structure remains stable.
[0136] (3) Efficient and flexible training: Adopting a hybrid training strategy of partially re-training the existing modules with LoRA and adding new modules saves video memory and training time compared to full-scale fine-tuning, and can flexibly adapt to different base models.
[0137] (4) Support for long videos: Solves the coherence problem of long shot generation and is applicable to a wider range of film and television production scenarios.
[0138] (5) The present invention proposes a digital human video generation method with camera control, allowing users to input a predefined camera trajectory file, and the model can generate a dynamic video that not only conforms to the reference character image and driving audio lip sync but also strictly follows the input camera trajectory movement.
[0139] (6) The present invention proposes a digital human video generation model based on the DiT (Diffusion Transformer) architecture. On the basis of the Wan2.1 base model, it innovatively designs the camera control injection model architecture and proposes a long video inference mechanism.
[0140] The above are merely preferred embodiments of the present invention and do not constitute any limitation on the present invention. Any equivalent substitutions or modifications made by those skilled in the art to the technical solutions and content disclosed in the present invention without departing from the scope of the present invention shall be deemed to have remained within the protection scope of the present invention.
Claims
1. A method for generating digital human videos with camera control, characterized in that, Includes the following steps: S1: Inject camera control into the model architecture design to generate a deep architecture from physical parameters to feature injection. The deep architecture includes the Plücker camera encoder, a diffusion transform-based camera control injection mechanism, and a post-adaptor module. S2: Set up a hybrid fine-tuning strategy, using a pre-trained digital human video generation model as the base model, and adopt a hybrid training strategy of full parameter training and LoRA fine-tuning. S3: Sets a time-step-aware dynamic window range strategy, which adopts a two-layer loop mechanism: the outer loop is the reverse denoising process of the diffusion model, and the inner loop is the sliding window processing of the entire video sequence at the current time step, which is used to support the generation of long videos with a duration exceeding the single inference window limit.
2. The digital human video generation method with camera control as described in claim 1, characterized in that, The English term for diffusion transformer is Diffusion Transformer, abbreviated as DiT.
3. The digital human video generation method with camera control as described in claim 2, characterized in that, The Plücker camera encoder includes the following features: Input and Function: Used to receive the camera extrinsic and intrinsic parameter matrices for each frame and convert them into feature tensors rich in geometric information and aligned with the video latent space; Gaze embedding generation: Calculate the gaze origin and gaze direction for each pixel in each frame of the image based on camera parameters, and generate Plücker embedding; Temporal dimension compression: In order to adapt to the temporal compression characteristics of the VAE latent space in the DiT model, this encoder adopts a lossless transformation strategy based on repetition and reshaping.
4. The digital human video generation method with camera control as described in claim 3, characterized in that, The algorithm for the Plücker camera encoder is as follows: The first frame of the input Plücker embed is copied 4 times, changing its timing length from 81 to 84. By using tensor reshaping and transpose operations, the data from four adjacent time frames are stitched together along the channel dimension. The final output is the camera feature tensor.
5. The digital human video generation method with camera control as described in claim 3, characterized in that, In the camera control injection mechanism based on diffusion converter, the camera control signal is deeply integrated into each DiT module to achieve block-by-block injection, thereby improving the model's compliance with camera control. The working principle of the camera control injection mechanism based on the diffusion converter is as follows: Each DiT module contains an independent camera embedded in a projector; The camera-embedded projector receives a 24-channel feature tensor from the Plücker camera encoder. The width and height of the tensor are downsampled to the same size as the noise latent variable using the PixelUnshuffle operator. Increase its dimension to the same number as the noise latent variable by using 2D convolution. Deep semantic features are extracted using residual blocks.
6. The digital human video generation method with camera control as described in claim 5, characterized in that, The post-adaptor module refines and adjusts the features processed by the camera control injection mechanism based on the diffusion transformer, enhancing the model's ability to coordinate digital lip alignment and camera motion.
7. The digital human video generation method with camera control as described in claim 1, characterized in that, Full-parameter training is used to train the Camera Embedding Projector and Adapter within each DiT block with full parameters, enabling them to quickly adapt to camera control tasks; LoRA fine-tuning is used to fine-tune the original Self-Attention layer using LoRA technology, preserving the original model's high-quality generation capabilities while adapting to new multimodal inputs.
8. The digital human video generation method with camera control as described in claim 1, characterized in that, The strategy for setting the dynamic window range for time-step awareness is as follows: Dynamic window offset: At each denoising time step, the starting position of the sliding window will be dynamically offset according to the preset step size, so that each frame is in a different relative position within the window at different denoising stages, thereby expanding the temporal context receptive field of each frame and eliminating the seam effect caused by fixed window division. Non-wrap and overlap mechanism: Dynamic offset is introduced while maintaining window overlap; Boundary adaptive constraints: Head protection: When the accumulated offset causes the effective length of the first segment to be less than a set threshold, the offset is automatically reset to prevent the window from being too short and affecting the generation quality; End-of-video padding: If the length of a segment at the end of a video is less than the minimum window requirement, the strategy will automatically extend its starting position forward to meet the minimum length constraint. This also prevents the window from being too short and affecting the generation quality.