A camera and light combined controllable 4D video generation method, device and equipment

Abstract

The application provides a 4D video generation method, device and equipment with controllable camera and illumination, relates to the technical field of computer vision and computer graphics, and aims to solve the problem that the existing model cannot simultaneously control the camera trajectory and the illumination condition. The method comprises the following steps: based on an input video, generating a dynamic point cloud and a sparse heavy-light point cloud, the dynamic point cloud is used for explicitly representing the geometric structure and motion information of the scene in the input video, and the sparse heavy-light point cloud is used for providing the illumination prior of the scene under the target illumination condition; according to a target camera trajectory, the dynamic point cloud and the sparse heavy-light point cloud are processed respectively to generate geometric prior information and illumination prior information aligned with a target view angle; the geometric prior information and the illumination prior information are encoded into visual tokens, and illumination query tokens are extracted from heavy-light key frames; denoising generation is performed by using the visual tokens and the illumination query tokens, and a target video consistent with the target camera trajectory and the target illumination condition is output.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and computer graphics, and in particular to a method, apparatus and device for generating 4D video with combined control of camera and lighting. Background Technology

[0002] In the field of AI-based video generation and editing, achieving high-fidelity and controllable generation of dynamic scenes is a current research hotspot.

[0003] Currently, while AI-based video generation technology has made significant progress in single-dimensional control, users who want to achieve complex visual effects involving perspective changes and lighting adjustments must undergo cumbersome multi-model concatenation. This is not only complex to operate but also prone to image quality loss and timing inconsistencies. Specifically, one camera trajectory control scheme achieves stable new perspective video generation by injecting camera pose parameters, but its architecture completely neglects lighting editing functions, resulting in generated videos that cannot adjust lighting conditions. Another video relighting technology, while able to change scene lighting based on text or image conditions, relies on the assumption of a fixed camera perspective, leading to severe geometric distortion once the perspective changes.

[0004] Therefore, no single existing model can simultaneously control both camera motion and lighting changes. Summary of the Invention

[0005] This invention provides a method, apparatus, and device for generating 4D video with joint control of camera and lighting, which solves the defect of existing models that cannot simultaneously control camera trajectory and lighting conditions, and realizes simultaneous joint control of camera trajectory and lighting conditions.

[0006] This invention provides a 4D video generation method with joint controllable camera and illumination, comprising the following steps: generating dynamic point clouds and sparse re-illuminated point clouds based on input video, wherein the dynamic point clouds are used to characterize the geometric structure and motion information of the scene in the input video, and the sparse re-illuminated point clouds are used to provide illumination priors for the scene under target illumination conditions; processing the dynamic point clouds and sparse re-illuminated point clouds respectively according to the target camera trajectory to generate geometric prior information and illumination prior information aligned with the target viewpoint; encoding the geometric prior information and illumination prior information into visual tokens, and extracting illumination query tokens from re-illuminated keyframes, wherein the re-illuminated keyframes are obtained by re-illuminating keyframes of the input video; using the visual tokens and illumination query tokens for denoising generation, and outputting a target video (i.e., a four-dimensional (4D) video) consistent with the target camera trajectory and target illumination conditions.

[0007] According to the present invention, a 4D video generation method with joint controllable camera and illumination is provided. The generation of dynamic point cloud includes: performing depth estimation on each frame of the input video to obtain a frame-by-frame depth map; and back-projecting the frame-by-frame depth map into a three-dimensional point cloud sequence based on camera parameters to form a dynamic point cloud.

[0008] According to the present invention, a 4D video generation method with camera and illumination controllable jointly is provided. The above-mentioned generation of sparse relit point cloud includes: selecting at least one key frame from the input video; performing relit processing on the key frame using target illumination conditions to obtain relit key frame; and back-projecting the relit key frame into point cloud based on camera parameters to form sparse relit point cloud.

[0009] According to the present invention, a 4D video generation method with camera and lighting controllable, wherein the target lighting conditions include at least one of text description, reference background image, high dynamic range (HDR) image or example image.

[0010] According to the present invention, a 4D video generation method with joint controllable camera and illumination is provided. The method involves processing dynamic point clouds and sparse, heavily illuminated point clouds according to the target camera trajectory to generate geometric prior information and illumination prior information aligned with the target viewpoint. The method includes: projecting and rendering the dynamic point cloud along the target camera trajectory to generate a geometrically aligned view and a first visibility mask as geometric prior information; and projecting and rendering the sparse, heavily illuminated point cloud along the target camera trajectory to generate an illumination cue view and a second visibility mask as illumination prior information.

[0011] According to the present invention, a camera- and illumination-controlled 4D video generation method is provided, wherein the above-mentioned extraction of illumination query tokens from re-illumination keyframes includes: encoding the re-illumination keyframes into latent space features using an encoder; and extracting the illumination query tokens from the latent space features using a query converter module.

[0012] According to the present invention, a camera-and-light-coordinated 4D video generation method is provided, wherein the above-mentioned denoising generation is performed using visual tokens and lighting query tokens to output a target video consistent with the target camera trajectory and target lighting conditions. The method includes: in a diffusion transformer, using a visual token as a query and a lighting query token as a key and value, performing a cross-attention operation to inject lighting information; performing multi-step denoising sampling based on the visual token after injecting lighting information; and reconstructing the target video through a decoder.

[0013] This invention also provides a 4D video generation device with joint controllable camera and illumination, comprising the following modules: a generation module, used to generate dynamic point clouds and sparse re-illuminated point clouds based on input video, wherein the dynamic point clouds are used to characterize the geometric structure and motion information of the scene in the input video, and the sparse re-illuminated point clouds are used to provide illumination priors for the scene under target illumination conditions; a rendering module, used to process the dynamic point clouds and sparse re-illuminated point clouds respectively according to the target camera trajectory to generate geometric prior information and illumination prior information aligned with the target viewpoint; an extraction module, used to encode the geometric prior information and illumination prior information into visual tokens, and extract illumination query tokens from re-illuminated keyframes, wherein the re-illuminated keyframes are obtained by re-illuminating keyframes of the input video; and a generation module, used to perform denoising generation using the visual tokens and illumination query tokens, and output a target video consistent with the target camera trajectory and target illumination conditions.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the camera and illumination jointly controllable 4D video generation method as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the camera and illumination jointly controllable 4D video generation method as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the camera and illumination jointly controllable 4D video generation method as described above.

[0017] The present invention provides a method, apparatus, and device for jointly controlling 4D video with camera and illumination. By generating dynamic point clouds representing scene geometric motion information and sparse, heavily illuminated point clouds providing target illumination priors, the invention achieves effective decoupling of geometry and illumination conditions. Furthermore, based on the target camera trajectory, the two types of point clouds are collaboratively processed to generate viewpoint-aligned geometric and illumination priors, ensuring the consistency of different control signals in the spatial dimension. Subsequently, by encoding the geometric and illumination priors into visual tokens and combining them with illumination query tokens extracted from heavily illuminated keyframes, a correlation mapping between illumination conditions and spatial geometry is established at the feature level. Finally, during the denoising and generation process, the synergistic effect of the visual tokens and illumination query tokens ensures that the generated video strictly follows the target camera trajectory to maintain geometric coherence and accurately responds to target illumination conditions to maintain illumination consistency, thus achieving joint control of camera trajectory and illumination conditions in a complete process. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating a 4D video generation method with combined camera and lighting control provided by the present invention.

[0020] Figure 2 This is a schematic diagram of the structure of a 4D video generation device with combined control of camera and illumination provided by the present invention.

[0021] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0023] The following is combined with Figure 1 The present invention describes a 4D video generation method with combined camera and illumination control.

[0024] Figure 1 This is a flowchart illustrating a camera-and-light-controlled 4D video generation method provided by the present invention, as shown below. Figure 1 As shown, the method includes the following:

[0025] S101. Generate dynamic point clouds and sparse heavy illumination point clouds based on the input video.

[0026] In the embodiments of this application, dynamic point clouds are used to explicitly represent the geometric structure and motion information of the scene in the input video.

[0027] In some embodiments, depth estimation can be performed on each frame of the input video to obtain a frame-by-frame depth map.

[0028] For example, a monocular depth estimation model (dense prediction transformer, DPT) can be used to process the input video frame by frame, and output a frame-by-frame depth map with the same size as the video frame.

[0029] For example, the input video can be a "desktop object placement" video with a resolution of 1280×720 and a frame rate of 30fps. The output frame-by-frame depth map has pixel values ​​ranging from 0 to 5 meters, representing the distance from the pixel to the camera.

[0030] Furthermore, after obtaining the frame-by-frame depth map, the frame-by-frame depth map can be back-projected into a 3D point cloud sequence based on the camera parameters to form a dynamic point cloud.

[0031] In this embodiment, camera parameters may include intrinsic and extrinsic parameters.

[0032] The intrinsic parameter can be the focal length f. x =1000 pixels, fᵧ=1000 pixels, principal point coordinates c x =640 pixels, cᵧ=360 pixels; the extrinsic parameters can be the camera's rotation matrix R and translation vector t for each frame; the extrinsic parameters can be recorded by the sensors built into the video shooting device or estimated by the simultaneous localization and mapping (SLAM) technique.

[0033] Specifically, back projection can be performed based on the pinhole camera model: for any pixel (x, y) in the frame-by-frame depth map, calculate the corresponding three-dimensional spatial coordinates, and then form a single-frame point cloud by combining the three-dimensional coordinates of all pixels in each frame. The single-frame point clouds are concatenated according to the video frame time sequence to obtain a complete dynamic point cloud sequence (e.g., 300 frames of video correspond to 300 single-frame point clouds, and each point cloud contains approximately 920,000 three-dimensional points).

[0034] The calculation of three-dimensional spatial coordinates can be as follows:

[0035] X=(xc x )×d / f x , Y=(y-cᵧ)×d / fᵧ, Z=d.

[0036] Where d is the depth value of the pixel.

[0037] Thus, this application constructs dynamic point clouds through frame-by-frame depth estimation and back-projection, providing an accurate and temporally coherent 3D geometric foundation for the subsequent generation process, effectively improving the realism and motion consistency of the scene structure from the new perspective.

[0038] In this embodiment, sparse heavily lit point clouds are used to provide lighting priors for the scene under target lighting conditions.

[0039] For example, the target lighting conditions may include at least one of a text description, a reference background image, an HDR image, or an example image.

[0040] Specifically, text descriptions refer to natural language input by the user to define the target lighting style, direction, or intensity. Examples include "warm yellow sidelight at sunset (800 nits brightness, 45° angle between the light source and the camera)" and "cool white top light in an operating room scene (6500K color temperature, uniform illumination)." The model can then transform these descriptions into lighting feature vectors using a text encoder (such as a contrastive language-image pretraining (CLIP) text encoder).

[0041] Specifically, the reference background image refers to a background image provided by the user that matches the scene type of the input video and contains target lighting information. For example, when the input video is a "bedroom scene," the user provides a "background image of a bedroom illuminated by a warm orange desk lamp." The model extracts the lighting distribution of this image (such as high brightness in the lamp area and low brightness in the shadow area) as a lighting reference.

[0042] Specifically, HDR images refer to images with a brightness range covering 0.1-10000 nits (far exceeding the 0-255 brightness range of ordinary red, green, blue, RGB images). For example, an HDR image containing "outdoor midday sunlight (10000 nits) + tree shade (500 nits)". HDR images can provide models with more realistic light intensity gradient information.

[0043] Specifically, the example image refers to a reference photograph provided by the user showing the same scene as the input video under the target lighting. For example, if the input video is a "desktop water glass," the user provides a photograph of a "water glass illuminated by candlelight (warm red light, light source on the left)." The model can directly learn the lighting details of this photograph (such as the left side of the water glass being brighter and the right side darker, and the reflection color of the glass wall being red).

[0044] Thus, by supporting multiple modalities of lighting condition input, such as text and images, this application enables users to drive lighting changes in a more intuitive and flexible way, greatly improving the system's practicality and ease of use in diverse application scenarios.

[0045] In some embodiments, at least one keyframe may be selected from the input video.

[0046] For example, the selection of keyframes can adopt a strategy that combines "fixed interval + adaptive filtering".

[0047] For example, for an input video of 300 frames, 10 candidate frames are initially selected at fixed intervals (e.g., every 30 frames). Then, the pixel displacement of adjacent candidate frames is calculated using an optical flow algorithm (e.g., the recurrent all-pairs field transforms (RAFT) optical flow model). Frames with displacement greater than 5 pixels (indicating drastic motion) are retained. At the same time, frames with significant exposure changes are selected based on the brightness difference between frames (e.g., the difference in the mean of RGB channels exceeds 20). Finally, 12 key frames are determined.

[0048] For example, if the input video has obvious occlusion, the proportion of the occluded area can be detected by depth map, and the frames before, during and after the occlusion can be supplemented as keyframes.

[0049] Specifically, taking a video of "tabletop objects being obscured by a hand" as an example, the proportions of the obscured area in frame 50 (before obscuration), frame 75 (during obscuration), and frame 100 (after obscuration) are 5%, 35%, and 8%, respectively. These three frames are then added as keyframes, and the total number of keyframes is adjusted to 15 to ensure coverage of the key states of scene movement, exposure, and obscuration.

[0050] Furthermore, for each keyframe, the keyframe is re-illuminated using the target lighting conditions to obtain a re-illuminated keyframe. Then, based on the camera parameters, the re-illuminated keyframe is back-projected into a point cloud to form a sparse re-illuminated point cloud.

[0051] For example, an existing image relighting model (such as a lighting process) can be used. Inputting the 50th keyframe (unobstructed desktop) and the text description "cool blue backlight (color temperature 10000K, brightness 500 nits)", the output is a relighting keyframe of that keyframe under cool blue backlight (dark left side of the desktop, bright right side, object shadows facing left). This operation is performed on each of the 15 keyframes to obtain 15 relighting keyframes.

[0052] Specifically, the backprojection process of the relighting keyframes is consistent with that of the dynamic point cloud: based on the same camera intrinsic parameters (f x 、fᵧ、c x The extrinsic parameters (R, t) of the corresponding keyframes are used to back-project each pixel (x, y) of the relighting keyframe into three-dimensional coordinates according to the depth value d (using the depth map of the original keyframe to ensure that the geometric position remains unchanged), forming 15 single-frame relighting point clouds. These are then concatenated in the keyframe time sequence to form a sparse relighting point cloud.

[0053] Thus, by relighting keyframes and constructing sparse point clouds, this application reduces computational overhead while providing the model with lighting samples derived from real images, ensuring the reliability of lighting priors and the fidelity of local details.

[0054] S102. Based on the target camera trajectory, process the dynamic point cloud and the sparse heavy illumination point cloud respectively to generate geometric prior information and illumination prior information aligned with the target viewpoint.

[0055] In this embodiment of the application, the target camera trajectory can be set by the user through an interactive interface.

[0056] For example, "Rotate the camera 30° clockwise around the table, keeping the camera height at 1.5 meters and the distance from the table 0.8 meters, generating a total of 30 frames of trajectory (corresponding to 30 frames of output video)". Each trajectory contains the extrinsic parameters of the camera for that frame (rotation matrix R1-R). 30 Translation vector t1-t 30 ).

[0057] In this embodiment, the geometric prior information includes a geometrically aligned view and a first visibility mask.

[0058] In some embodiments, the dynamic point cloud can be projected and rendered along the target camera trajectory to generate a geometrically aligned view and a first visibility mask.

[0059] For example, projection rendering can be performed using the PyTorch 3D graphics pipeline to project the poses of 300 single-frame point clouds of a dynamic point cloud along 30 trajectories.

[0060] Specifically, using the 15th trajectory pose (R) 15 t 15 Taking a 15° rotation viewpoint as an example, the 3D points (desktop objects) in the 100th frame of the dynamic point cloud are processed using R... 15 Rotation, t 15 After translation, the image is projected onto the camera's imaging plane along the trajectory to obtain a geometrically aligned view with a resolution of 1280×720 (the side of the desktop object as seen from a 15° angle).

[0061] For example, the first visibility mask is a binary image with the same size as the view. Pixel value 1 indicates that there is a valid point cloud projection at that location (no occlusion), and pixel value 0 indicates that there is no point cloud at that location (or it is occluded by other points).

[0062] Optionally, a depth map can also be generated simultaneously to record the Z value of the projection point for subsequent occlusion boundary optimization, as well as to generate a normal map and calculate the surface normal vector of the point cloud, such as the normal of the table being vertically upward and the normal of the cup being along the tangent direction of the cup wall.

[0063] In this embodiment, the prior lighting information includes a lighting cue view and a second visibility mask.

[0064] In some embodiments, sparse heavily lit point clouds can be projected and rendered along the target camera trajectory to generate a lighting cue view and a second visibility mask.

[0065] For example, sparse relit point clouds (such as 15 single-frame relit point clouds) can be projected along the aforementioned 30 target camera trajectories. For instance, the pose of the 50th frame relit point cloud (cool blue backlight) can be projected along the 15th trajectory.

[0066] Specifically, the projection process is consistent with the dynamic point cloud. The 3D points in the relighting point cloud carry the RGB color information (cool blue tone) of the relighting key frame. After projection, a lighting cue view is obtained (the "brightness and darkness distribution of desktop objects under cool blue backlight" seen from a 15° perspective, with the right side of the object being bright and the left side being dark).

[0067] It should be noted that the second visibility mask is generated using the same logic as the first visibility mask. It is used to mark the valid projection area (pixel value 1) and invalid area (pixel value 0) in the lighting cues view, ensuring that the lighting cues only apply to the visible part of the scene and avoiding interference from invalid areas in lighting generation.

[0068] Thus, by projecting point clouds into concrete views and masks, this application transforms abstract geometric and lighting priors into visual cues that can be directly utilized by the model, enhancing spatial alignment capabilities and providing crucial information for handling occlusion boundaries.

[0069] S103. Encode the geometric prior information and the lighting prior information into a visual token, and extract the lighting query token from the relighting keyframe.

[0070] Among them, the relighting keyframe is obtained by relighting the keyframes of the input video.

[0071] In some embodiments, the geometrically aligned view, the first visibility mask, the lighting cue view, and the second visibility mask can be encoded as visual tokens.

[0072] For example, a pre-trained vision transformer (ViT-B / 16) can be used as an encoder to concatenate the geometrically aligned view (e.g., 3-channel RGB), the first visibility mask (e.g., 1-channel binary image), the illumination cue view (e.g., 3-channel RGB), and the second visibility mask (e.g., 1-channel binary image) into an 8-channel feature map (e.g., resolution 1280×720). The feature map is then divided into 16×16 pixel image patches, resulting in a total of (1280 / 16)×(720 / 16)=50×45=2250 patches.

[0073] Specifically, each image patch can be transformed into a 768-dimensional vector through the embedding layer of ViT (embedding dimension, Embedd_dim = 768), and then input into ViT's 12-layer Transformer encoder (including multi-head self-attention and feedforward network), outputting a feature sequence of length 2250 and dimension 768. This sequence is the visual token, and each token corresponds to the comprehensive features (geometry + illumination + visibility) of a local region (16×16 pixels) of the original feature map.

[0074] Furthermore, an encoder can be used to encode relighting keyframes into latent space features.

[0075] For example, the encoder can use a pre-trained variational autoencoder (VAE) (input resolution 224×224, latent space dimension 512), which scales 15 relighting keyframes (such as the 50th frame of cool blue backlight keyframe) to 224×224 before inputting them, and outputs 512-dimensional latent space features (including the color, direction and intensity information of the illumination).

[0076] Furthermore, the illumination query token can be extracted from the latent space features using the query converter module.

[0077] For example, the query transformer module can be a query transformer containing 16 learnable query vectors (each vector has a dimension of 512). During the pre-training phase, it learns to extract core lighting attributes by matching latent space features of a large number of relit keyframes with cross-attention.

[0078] Specifically, the 512-dimensional latent space features of the relighting keyframe can be input into the Q-Former. The 16 query vectors of the Q-Former are used to calculate attention weights through cross-attention (the query is the Q-Former query vector, and the key and value are latent space features). After weighted summation, 16 512-dimensional feature vectors are output. This vector is the lighting query token, and each token corresponds to a lighting attribute (e.g., token 1 corresponds to color temperature, token 2 corresponds to brightness, and token 3 corresponds to illumination direction).

[0079] Thus, this application distills a compact global illumination representation from the relighting keyframes through two-level feature processing of the encoder and the query converter, providing a core control signal for injecting a stable and consistent lighting style throughout the video sequence.

[0080] S104. Use visual tokens and illumination query tokens to perform denoising and generate a target video that matches the target camera trajectory and target illumination conditions.

[0081] In some embodiments, a cross-attention operation can be performed in the diffusion transformer, using a visual token as a query and an illumination query token as a key and value, to inject illumination information.

[0082] Among them, the diffusion transformer is a lightweight diffusion transformer (i.e., the global illumination control layer in a diffusion transformer).

[0083] For example, the diffusion model can be based on the diffusionTransformer XL / 2 architecture (DiT-XL / 2), with the input being noisy initial features (consistent with the visual token dimension: 2250×768), and the Light-DiT layer serving as the core layer of this architecture, containing a multi-head cross-attention module.

[0084] Specifically, the visual token (2250×768) can be used as the query (Q) for cross-attention, and the illumination query token (16×512) can be mapped to 768 dimensions through a linear layer, serving as the key (K) and value (V) for cross-attention. The attention score is then calculated as: Attention(Q, K, V) = Softmax(Q×Kᵀ / )×V.

[0085] in, As a scaling factor (to alleviate the curse of dimensionality), the Softmax function normalizes the attention score to 0-1, and finally outputs a 2250×768 visual token that incorporates lighting information (each visual token carries global lighting properties).

[0086] Furthermore, multi-step denoising sampling is performed based on the visual token after injecting illumination information, and the target video is reconstructed through a decoder.

[0087] For example, denoising sampling can be performed in 1000 steps, with the fusion token updated through the Light-DiT layer at each step: initially a random noise token, the first step uses the visual token of the first frame plus the illumination query token for denoising, the second step uses the token of the second frame for denoising, until a noise-free fusion token is obtained in the 1000th step.

[0088] In this embodiment of the application, the decoder may be a VAE decoder (matching the VAE encoder of S103), which maps the 2250×768 fusion token to 2250×(16×16×3) pixel features (corresponding to the RGB values ​​of 16×16 pixels) through layer normalization (LayerNorm) and linear layer, and then reassembles the pixel features into a 1280×720 RGB image frame.

[0089] Specifically, the 30 denoised image frames can be concatenated according to the target camera trajectory to obtain a target video with a resolution of 1280×720 and a frame rate of 30fps. This video meets both the viewing angle requirement of "rotating 30° around the table" and the lighting requirement of "cool blue backlight", and the geometric structure of the objects (such as the height of the cup and the shape of the table) is consistent with the original video, with no flickering in the lighting.

[0090] Optionally, three modes can be selected during the inference process.

[0091] In one example, if the "Joint Controllable Mode" is selected, the complete process described above (geometry + lighting control) is enabled.

[0092] In another example, if "Camera Control Only Mode" is selected, sparse heavy illumination point cloud generation, illumination cue view and second visibility mask encoding, and illumination query token extraction are disabled, and only dynamic point cloud, geometrically aligned view and first visibility mask are retained to generate a new perspective video.

[0093] In another example, if "Relighting Only Mode" is selected, the target camera trajectory is set to constant (e.g., R is the identity matrix, t is 0, i.e., the original viewpoint), and only the lighting cues and lighting query tokens are retained to generate the original viewpoint relighting video.

[0094] Thus, this application achieves deep fusion of geometric visual features and global illumination features by using a cross-attention mechanism in the diffusion transformer, thereby collaboratively generating video frames that have both geometric correctness and illumination consistency during the denoising process.

[0095] In the camera- and lighting-controlled 4D video generation method provided in this application embodiment, effective decoupling of geometry and lighting conditions is achieved by generating dynamic point clouds that represent scene geometric motion information and sparse, heavily lit point clouds that provide target lighting priors. Then, based on the target camera trajectory, the two types of point clouds are collaboratively processed to generate viewpoint-aligned geometric and lighting priors, ensuring the coordination and consistency of different control signals in the spatial dimension. Subsequently, by encoding the geometric and lighting priors into visual tokens and combining them with lighting query tokens extracted from heavily lit keyframes, a correlation mapping between lighting conditions and spatial geometry is established at the feature level. Finally, during the denoising generation process, the synergistic effect of visual tokens and lighting query tokens ensures that the generated video can strictly follow the target camera trajectory to maintain geometric coherence and accurately respond to target lighting conditions to maintain lighting consistency, thus achieving joint control of camera trajectory and lighting conditions in a complete process.

[0096] Optionally, when the input video is a long video sequence (e.g., 1000 frames), the illumination query token can be periodically extracted and updated based on the generated target video to suppress cumulative illumination drift during the generation process.

[0097] For example, the sliding window size can be set to 50 frames (i.e., a lighting calibration is performed after every 50 frames of video are generated), and the initially extracted lighting query token (from the input video keyframe) is used when generating frames 1-50; after the 50th frame of the target video is generated, the 50th frame is used as the calibration frame.

[0098] Specifically, subsequent calibration operations may include:

[0099] 1) Based on the camera parameters (intrinsic parameters consistent with S101, extrinsic parameters of the target trajectory in frame 50), back-project the back-correction frame to obtain the back-correction point cloud (the three-dimensional coordinate calculation method is the same as S101).

[0100] 2) Project and render the back-calibration point cloud along the current target camera trajectory (external parameters of frame 51) to obtain the back-calibration view (resolution 1280×720, including the lighting status of frame 50).

[0101] 3) Input the backtracking view into the VAE encoder to obtain the 512-dimensional backtracking latent space features, and then extract the "calibration illumination query token" again through Q-Former.

[0102] 4) Replace the illumination query token in the generation process of frames 51-100 with the calibration token to calibrate the cross-attention operation of the Light-DiT layer (e.g., if the original token causes the brightness of frame 45 to be slightly lower, the calibration token adjusts the brightness weight so that the brightness of frames 51-100 is consistent with that of frame 50).

[0103] Thus, this application, for long video sequences, can dynamically calibrate the lighting state by periodically updating the lighting query token, effectively suppressing lighting drift (such as gradual decrease in brightness) or flicker (abrupt changes in brightness between frames) caused by error accumulation, and ensuring the visual coherence of long-time sequence generation.

[0104] Optionally, to achieve large-scale training of the model, this application can also construct training samples through the Light-Syn data pipeline.

[0105] For example, a real-world scene video (such as a "people walking around in an office" video with a resolution of 1920×1080, containing 5 viewpoints and 4 lighting conditions) can be acquired as the target video Vt. Then, degradation processing is performed on Vt, such as adding Gaussian noise (standard deviation 0.03), image compression standard (joint photographic experts group, JPEG) compression (quality 60%), and exposure drift (random fluctuation of brightness between frames ±15%), to obtain the degraded video Vs. The degradation transformation parameters (noise type, compression quality, exposure fluctuation range) are recorded.

[0106] Specifically, inverse mapping can be performed based on degradation transformation parameters. For example, the depth map of the degradation video can be restored by using a denoising algorithm (such as BM3D). The dynamic point cloud (geometric cues) and sparse heavy illumination point cloud (illumination cues) of the target video can be aligned to the resolution and brightness domain of the degradation video to construct paired samples of "degraded video Vs-geometric cues + illumination cues", covering a total of 20 combinations of "5 viewpoints × 4 illumination".

[0107] Thus, this application can meet the training requirements without actually collecting multi-view, multi-light video, thereby improving the robustness of the model.

[0108] The following describes the camera-and-light-controlled 4D video generation apparatus provided by the present invention. The camera-and-light-controlled 4D video generation apparatus described below and the camera-and-light-controlled 4D video generation method described above can be referred to in correspondence.

[0109] Figure 2 This is a structural diagram of a 4D video generation device with jointly controllable camera and lighting, provided in an embodiment of this application. The 4D video generation device 200 with jointly controllable camera and lighting includes: a generation module 201, a rendering module 202, an extraction module 203, and an output module 204.

[0110] The system comprises: a generation module 201, used to generate dynamic point clouds and sparse re-illuminated point clouds based on the input video; the dynamic point clouds are used to represent the geometric structure and motion information of the scene in the input video, and the sparse re-illuminated point clouds are used to provide lighting priors for the scene under the target lighting conditions; a rendering module 202, used to process the dynamic point clouds and sparse re-illuminated point clouds respectively according to the target camera trajectory to generate geometric prior information and lighting prior information aligned with the target viewpoint; an extraction module 203, used to encode the geometric prior information and lighting prior information into visual tokens, and extract lighting query tokens from the re-illuminated keyframes, which are obtained by re-illuminating keyframes of the input video; and a generation module 204, used to perform denoising generation using the visual tokens and lighting query tokens, and output a target video consistent with the target camera trajectory and target lighting conditions.

[0111] In some embodiments, the generation module 201 is specifically used to: perform depth estimation on each frame of the input video to obtain a frame-by-frame depth map; and back-project the frame-by-frame depth map into a three-dimensional point cloud sequence based on camera parameters to form a dynamic point cloud.

[0112] In some embodiments, the generation module 201 is specifically used to: select at least one key frame from the input video; perform relighting processing on the key frame using target lighting conditions to obtain a relighting key frame; and backproject the relighting key frame into a point cloud based on camera parameters to form a sparse relighting point cloud.

[0113] In some embodiments, the target lighting conditions described above include at least one of a text description, a reference background image, an HDR image, or an example image.

[0114] In some embodiments, the rendering module 202 described above is specifically used to: project and render the dynamic point cloud along the target camera trajectory to generate a geometrically aligned view and a first visibility mask as geometric prior information; and project and render the sparsely lit point cloud along the target camera trajectory to generate a lighting cue view and a second visibility mask as lighting prior information.

[0115] In some embodiments, the extraction module 203 is specifically used to: encode the relighting keyframe into latent space features using an encoder; and extract the illumination query token from the latent space features using a query converter module.

[0116] In some embodiments, the output module 204 is specifically used to: in the diffusion transformer, perform a cross-attention operation to inject illumination information using a visual token as a query and an illumination query token as a key and value; perform multi-step denoising sampling based on the visual token after the illumination information is injected, and reconstruct the target video through a decoder.

[0117] In the 4D video generation apparatus with joint controllable camera and lighting provided in this application embodiment, effective decoupling of geometry and lighting conditions is achieved by generating dynamic point clouds that represent scene geometric motion information and sparse heavily lit point clouds that provide target lighting priors. Then, based on the target camera trajectory, the two types of point clouds are processed collaboratively to generate viewpoint-aligned geometric and lighting priors, ensuring the coordination and consistency of different control signals in the spatial dimension. Subsequently, by encoding the geometric and lighting priors into visual tokens and combining them with lighting query tokens extracted from heavily lit keyframes, a correlation mapping between lighting conditions and spatial geometry is established at the feature level. Finally, in the denoising generation process, through the synergistic effect of visual tokens and lighting query tokens, the generated video can strictly follow the target camera trajectory to maintain geometric coherence and accurately respond to the target lighting conditions to maintain lighting consistency, thereby realizing joint control of camera trajectory and lighting conditions in a complete process.

[0118] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3As shown, the electronic device 300 may include: a processor 310, a communication interface 320, a memory 320, and a communication bus 330, wherein the processor 310, the communication interface 320, and the memory 320 communicate with each other through the communication bus 330. The processor 310 can call logic instructions in the memory 320 to execute a camera- and lighting-controlled 4D video generation method. This method includes: generating dynamic point clouds and sparse re-illuminated point clouds based on the input video; the dynamic point clouds characterize the geometric structure and motion information of the scene in the input video, and the sparse re-illuminated point clouds provide lighting priors for the scene under target lighting conditions; processing the dynamic point clouds and sparse re-illuminated point clouds separately according to the target camera trajectory to generate geometric and lighting prior information aligned with the target viewpoint; encoding the geometric and lighting prior information into visual tokens and extracting lighting query tokens from re-illuminated keyframes; the re-illuminated keyframes are obtained by re-illuminating keyframes of the input video; and using the visual tokens and lighting query tokens for denoising generation to output a target video consistent with the target camera trajectory and target lighting conditions.

[0119] Furthermore, the logical instructions in the aforementioned memory 320 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0120] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the camera-and-light jointly controllable 4D video generation method provided by the above methods. The method includes: generating dynamic point clouds and sparse re-illuminated point clouds based on input video, wherein the dynamic point clouds are used to characterize the geometric structure and motion information of the scene in the input video, and the sparse re-illuminated point clouds are used to provide lighting priors of the scene under target lighting conditions; processing the dynamic point clouds and sparse re-illuminated point clouds respectively according to the target camera trajectory to generate geometric prior information and lighting prior information aligned with the target viewpoint; encoding the geometric prior information and lighting prior information into visual tokens, and extracting lighting query tokens from re-illuminated keyframes, wherein the re-illuminated keyframes are obtained by re-illuminating keyframes of the input video; and using the visual tokens and lighting query tokens to perform denoising generation and output a target video consistent with the target camera trajectory and target lighting conditions.

[0121] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a camera- and illumination-controlled 4D video generation method provided by the methods described above. This method includes: generating a dynamic point cloud and a sparse re-illuminated point cloud based on an input video, wherein the dynamic point cloud is used to characterize the geometric structure and motion information of the scene in the input video, and the sparse re-illuminated point cloud is used to provide illumination priors for the scene under target illumination conditions; processing the dynamic point cloud and the sparse re-illuminated point cloud respectively according to the target camera trajectory to generate geometric prior information and illumination prior information aligned with the target viewpoint; encoding the geometric prior information and illumination prior information into visual tokens, and extracting illumination query tokens from re-illuminated keyframes, wherein the re-illuminated keyframes are obtained by re-illuminating keyframes of the input video; and using the visual tokens and illumination query tokens for denoising generation to output a target video consistent with the target camera trajectory and target illumination conditions.

[0122] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0123] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for 4D video generation with controllable camera and lighting, characterized in that, include: Based on the input video, dynamic point clouds and sparse heavy illumination point clouds are generated. The dynamic point clouds are used to explicitly represent the geometric structure and motion information of the scene in the input video, and the sparse heavy illumination point clouds are used to provide illumination priors of the scene under the target illumination conditions. The dynamic point cloud is projected and rendered along the target camera trajectory to generate a geometrically aligned view and a first visibility mask as geometric prior information. The sparse, heavily lit point cloud is projected and rendered along the trajectory of the target camera to generate a lighting cue view and a second visibility mask as lighting prior information. The geometric prior information and the illumination prior information are encoded into a visual token, and the illumination query token is extracted from the re-illumination keyframe. The re-illumination keyframe is obtained by re-illuminating the keyframe of the input video. In the diffusion converter, the visual token is used as a query and the illumination query token is used as a key and value to perform a cross-attention operation to inject illumination information. Multi-step denoising sampling is performed based on the visual token after injecting illumination information, and the target video is reconstructed through the decoder to obtain a target video that is consistent with the target camera trajectory and the target illumination conditions.

2. The method according to claim 1, characterized in that, The generation of the dynamic point cloud includes: Depth estimation is performed on each frame of the input video to obtain a frame-by-frame depth map; The frame-by-frame depth map is back-projected into a three-dimensional point cloud sequence based on camera parameters to form the dynamic point cloud.

3. The method according to claim 1, characterized in that, Generating sparsely illuminated point clouds includes: Select at least one keyframe from the input video; The keyframe is re-illuminated using the target lighting conditions to obtain the re-illuminated keyframe. The re-illuminated keyframes are back-projected into point clouds based on camera parameters to form the sparse re-illuminated point cloud.

4. The method according to claim 1 or 3, characterized in that, The target lighting conditions include at least one of the following: a text description, a reference background image, a high dynamic range (HDR) image, or an example image.

5. The method according to claim 1, characterized in that, The extraction of the illumination query token from the relighting keyframe includes: The relighting keyframes are encoded into latent space features using an encoder; The illumination query token is extracted from the latent space features using the query converter module.

6. A 4D video generation device with combined camera and lighting controllability, characterized in that, The device includes: A generation module is used to generate dynamic point clouds and sparse heavy illumination point clouds based on input video. The dynamic point clouds are used to characterize the geometric structure and motion information of the scene in the input video, and the sparse heavy illumination point clouds are used to provide illumination priors of the scene under target illumination conditions. The rendering module is used for: The dynamic point cloud is projected and rendered along the target camera trajectory to generate a geometrically aligned view and a first visibility mask as geometric prior information. The sparse, heavily lit point cloud is projected and rendered along the trajectory of the target camera to generate a lighting cue view and a second visibility mask as lighting prior information. The extraction module is used to encode the geometric prior information and the illumination prior information into a visual token, and extract the illumination query token from the re-illumination keyframe, wherein the re-illumination keyframe is obtained by re-illuminating the keyframe of the input video. Generate modules for: In the diffusion converter, the visual token is used as a query and the illumination query token is used as a key and value to perform a cross-attention operation to inject illumination information. Multi-step denoising sampling is performed based on the visual token after injecting illumination information, and the target video is reconstructed through the decoder to obtain a target video that is consistent with the target camera trajectory and the target illumination conditions.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the camera and lighting jointly controllable 4D video generation method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the camera and lighting jointly controllable 4D video generation method as described in any one of claims 1 to 5.

Images