A high dynamic range illumination estimation method, device and storage medium

Abstract

The present application relates to a kind of high dynamic range illumination estimation method, device and storage medium, utilize the spatiotemporal generation prior of video generation model, by constructing "in-video drawing" task, from ordinary low dynamic range video sequence, restore time sequence consistent and physically realistic high dynamic range environment map.The input video is applied mask and combined with specific text prompt, guide model to generate virtual light probe containing surrounding environment reflection information in picture center;Variational autoencoder of high dynamic range perception is constructed, by introducing learnable tone mapping and inverse tone mapping adapter, so that video diffusion model can directly understand and output high dynamic range illumination characteristics in logarithmic domain latent space.The present application can output high-quality dynamic high dynamic range environment sequence, when processing dynamic scene, fast camera motion and complex occlusion, it has stronger robustness, applicable to augmented reality, virtual object implantation and scene relighting etc.Application.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and graphics technology, specifically relating to a high dynamic range illumination estimation method, device, and storage medium. Background Technology

[0002] Lighting estimation, as a key method for achieving virtual-real fusion in virtual content creation, has become an important research direction in the fields of Virtual Reality (VR) and Augmented Reality (AR). Accurate lighting estimation is fundamental to achieving realistic augmented reality, virtual object insertion, and scene relighting.

[0003] Traditional physics-based methods typically assume that lighting is static or simplified, making it difficult to handle complex dynamic scenes. Existing deep learning-based methods mostly focus on lighting estimation for single images, which leads to a lack of temporal consistency in lighting estimation between frames when processing video, resulting in flickering.

[0004] Furthermore, while existing video generation models can produce realistic lighting effects, their internal representations are typically optimized for low dynamic range (LDR) content and cannot directly output physically realistic high dynamic range (HDR) lighting information. Simply using a video model as an LDR-to-HDR translator often ignores the model's inherent generative priors. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a high dynamic range illumination estimation method, device and storage medium. It utilizes the inherent illumination understanding capability of the video generation model to realize illumination estimation by reconstructing the task form. The illumination estimation task is reconstructed into a "video inpainting" task, that is, a virtual "light probe" is inserted into the input video, which forces the video generation model to generate physically reliable reflections on the probe surface according to the surrounding spatiotemporal context, and then infer the ambient illumination.

[0006] The technical problem solved by this invention is achieved through the following technical solution:

[0007] A high dynamic range illumination estimation method, the steps of which are as follows:

[0008] S1. Obtain the low dynamic range (LDR) video sequence to be processed;

[0009] S2. Construct video in-view task input, including applying a mask to the low dynamic range to occlude a specific area, and generating text prompts to describe the virtual light probe;

[0010] S3. Using a pre-trained and finely tuned video diffusion model, based on the video in-painting task input, a virtual light probe video frame containing reflection information of the surrounding scene is generated in the occluded area; the video diffusion model directly outputs HDR latent features by learning the latent spatial representation of high dynamic range (HDR);

[0011] S4. Use an HDR-sensing decoder to reconstruct the HDR latent features into HDR video frames;

[0012] S5. Extract reflection information from the virtual light probe region of the HDR video frame and generate a time-consistent dynamic HDR environment map sequence through projection transformation.

[0013] Furthermore, the construction and training of the HDR latent spatial representation and HDR perceptual decoder in S3 and S4 are specifically as follows:

[0014] S2.1 Construct a variational autoencoder (VAE) containing an encoder and a decoder.

[0015] S2.2 Introduce a learnable tone mapping adapter and an inverse tone mapping adapter; the tone mapping adapter is used to convert the input linear HDR video frame into a logarithmic domain representation, and then input it into the encoder to generate latent features;

[0016] S2.3 The decoder is used to decode the latent features into logarithmic domain features, and then transform them back into the linear HDR domain by the inverse tone mapping adapter;

[0017] S2.4. A phased fine-tuning strategy is adopted: In the first phase, the VAE parameters are frozen and only the tone mapping adapter is trained; in the second phase, the decoder and adapter are fine-tuned together to adapt the model to the HDR data distribution.

[0018] Furthermore, the calculation formula for the tone mapping adapter includes performing a logarithmic transformation on the input signal and adjusting it through convolutional layers, learnable scaling factors, and bias terms to preserve illumination cues.

[0019] Moreover, the training process of the S3 video diffusion model adopts a low-rank adaptation (LoRA) fine-tuning strategy; the video diffusion model is fine-tuned using a pairwise dataset containing LDR videos, corresponding HDR ground truth videos (including virtual light probes) and text descriptions, so that it aligns with the distribution of the HDR latent space while maintaining the original LDR video generation prior.

[0020] Furthermore, the specific area in S2 is the central area of ​​the video image, the text prompt describes a sphere with perfect specular reflection properties, and the projection transformation in S5 is to unfold the spherical reflection into an equidistant cylindrical projection.

[0021] Furthermore, the system is trained using a synthetic dataset that includes dynamic lighting variations. This dataset is generated by rendering virtual scenes in a 3D rendering engine using panoramic video as dynamic environment textures and applying random exposure enhancements.

[0022] A high dynamic range illumination estimation device, comprising:

[0023] The data acquisition module is used to acquire the LDR video sequence to be processed;

[0024] The task building module is used to generate a masked video input and a text prompt describing the virtual light probe;

[0025] The generation module contains a video diffusion model with an integrated tone mapping adapter for generating HDR format virtual light probe video content in the masked area.

[0026] The post-processing module is used to extract and expand reflection information from the generated content and output a dynamic HDR environment map.

[0027] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method.

[0028] The advantages and beneficial effects of this invention are as follows:

[0029] 1. This invention designs an HDR-aware variational autoencoder (VAE) that maps the input linear HDR video to the logarithmic latent space by introducing a learnable tone mapping adapter, ensuring that the model can handle extreme changes in light intensity without losing details.

[0030] 2. This invention takes the occlusion of the central region of the LDR video as a conditional input, uses a pre-trained video diffusion Transformer model, and combines it with a text prompt describing a "perfectly mirror-reflecting sphere" to generate a chrome sphere of HDR content in the occluded area.

[0031] 3. This invention uses low-rank adaptation (LoRA) technology to fine-tune the video diffusion model to adapt it to the distribution of the HDR latent space, while retaining the spatiotemporal consistency prior learned by the pre-trained model on large-scale video data.

[0032] 4. The present invention extracts the generated chrome sphere region from the HDR video frame and generates an HDR environment map with equidistant cylindrical projection through back projection.

[0033] 5. This invention constructs a large-scale synthetic dataset (V-LITESET) containing video pairs with dynamic lighting changes generated from panoramic videos for model training.

[0034] 6. This invention utilizes the spatiotemporal attention mechanism of the video generation model to generate environment maps that are highly coherent in time, thus avoiding the flickering problem of single-frame methods.

[0035] 7. This invention, through an improved HDR VAE, enables the model to directly infer in the HDR domain and output physically true brightness information, rather than pseudo-HDR.

[0036] 8. This invention transforms illumination estimation into a generation task, enabling it to handle extreme cases such as fast camera movement and complex occlusion. Attached Figure Description

[0037] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0038] The present invention will be further described in detail below through specific embodiments. The following embodiments are merely descriptive and not limiting, and should not be used to limit the scope of protection of the present invention.

[0039] This invention proposes a dynamic high dynamic range (HDR) illumination estimation method based on a video generation model, used to recover temporally consistent and physically realistic HDR environment maps from ordinary low dynamic range (LDR) video sequences. This method takes monocular LDR video as input, constructs an "in-video rendering" task, combines it with specific text prompts, and utilizes the spatiotemporal priors of the video generation model to infer ambient illumination, outputting a high-quality dynamic HDR environment map sequence.

[0040] To achieve the above functions, this invention constructs an HDR-aware variational autoencoder (VAE) and an adaptively fine-tuned video diffusion model architecture. Its core lies in introducing a learnable tone mapping adapter to process HDR information in the logarithmic latent space and maintaining the spatiotemporal consistency of the generative model through a low-rank adaptation (LoRA) strategy. This invention also provides a synthetic dataset, V-LITESET, for training this model, containing rich video pairs with dynamic lighting variations, improving the model's robustness in complex scenes. The overall technical solution is as follows: Figure 1 As shown.

[0041] A high dynamic range illumination estimation method, the innovation of which lies in the following steps:

[0042] S1. Obtain the low dynamic range (LDR) video sequence to be processed;

[0043] S2. Construct video in-view task input, including applying a mask to the low dynamic range to occlude a specific area, and generating text prompts to describe the virtual light probe;

[0044] S3. Using a pre-trained and finely tuned video diffusion model, based on the video in-painting task input, a virtual light probe video frame containing reflection information of the surrounding scene is generated in the occluded area; the video diffusion model directly outputs HDR latent features by learning the HDR latent spatial representation.

[0045] S4. Use an HDR-sensing decoder to reconstruct the HDR latent features into HDR video frames;

[0046] S5. Extract reflection information from the virtual light probe region of the HDR video frame and generate a time-consistent dynamic HDR environment map sequence through projection transformation.

[0047] The construction and training of the HDR latent spatial representation and HDR perceptual decoder are as follows:

[0048] S2.1 Construct a variational autoencoder (VAE) containing an encoder and a decoder.

[0049] S2.2 Introduce a learnable tone mapping adapter and an inverse tone mapping adapter; the tone mapping adapter is used to convert the input linear HDR video frame into a logarithmic domain representation, and then input it into the encoder to generate latent features;

[0050] S2.3 The decoder is used to decode the latent features into logarithmic domain features, and then transform them back into the linear HDR domain by the inverse tone mapping adapter;

[0051] S2.4. A phased fine-tuning strategy is adopted: In the first phase, the VAE parameters are frozen and only the tone mapping adapter is trained; in the second phase, the decoder and adapter are fine-tuned together to adapt the model to the HDR data distribution.

[0052] The calculation formula for the tone mapping adapter includes performing a logarithmic transformation on the input signal and adjusting it through convolutional layers, a learnable scaling factor, and a bias term to preserve illumination cues.

[0053] The training process of the S3 video diffusion model adopts a low-rank adaptation (LoRA) fine-tuning strategy. The video diffusion model is fine-tuned using a pairwise dataset containing LDR videos, corresponding HDR ground truth videos (including virtual light probes), and text descriptions, so that it aligns with the distribution of the HDR latent space while maintaining the original LDR video generation prior.

[0054] The specific area in S2 is the central area of ​​the video image, the text prompt describes a sphere with perfect specular reflection properties, and the projection transformation in S5 is to unfold the spherical reflection into an equidistant cylindrical projection.

[0055] This invention utilizes a synthetic dataset containing dynamic lighting changes for training. This dataset is generated by rendering virtual scenes in a 3D rendering engine using panoramic video as a dynamic environment texture and applying random exposure enhancement.

[0056] A high dynamic range illumination estimation device, its innovation lies in: including:

[0057] (1) Video in-video rendering task construction module:

[0058] First, a low dynamic range (LDR) video sequence is acquired as the basic input. To infer ambient lighting using the generative model, a video in-frame rendering task is constructed. Specifically, the system automatically applies a mask to the central region of the input video frame to occlude it, while simultaneously generating a text prompt describing a "perfectly specular reflective sphere." This step aims to force the video generation model to "draw" a virtual light probe that reflects the surrounding environment in the occluded area, based on the spatiotemporal context information of the unoccluded area and utilizing its internally learned physical laws. This approach transforms the lighting estimation problem into a video completion problem, fully leveraging the video model's understanding of scene geometry and lighting relationships.

[0059] (2) HDR-Perceptual Variational Autoencoder (HDR-VAE):

[0060] This module addresses the limitation of traditional video generation models, which can only process LDR data. The system constructs a variational autoencoder (VAE) comprising an encoder and a decoder, and introduces a learnable tone mapping adapter and an inverse tone mapping adapter. The tone mapping adapter, located before the encoder, uses logarithmic transformation combined with convolutional layers, a learnable scaling factor, and a bias term to convert the input linear HDR video frames into a logarithmic domain suitable for model processing, thereby compressing the dynamic range while preserving physical cues in the highlights. The inverse tone mapping adapter, located after the decoder, is responsible for accurately restoring the decoded logarithmic domain features back to the linear HDR domain.

[0061] This module enables the video diffusion model to perform feature learning and inference directly in the HDR latent space, rather than simply outputting pixel colors, thus establishing an accurate representation of physical illumination information.

[0062] (3) Video Diffusion Generation Network Based on Stream Matching

[0063] This module utilizes a pre-trained stream-matching video diffusion Transformer model to perform generation tasks within the latent space constructed by HDR-VAE. Inputs include latent features of the masked LDR video and text cue word features. The model leverages its spatiotemporal attention mechanism, pre-trained on large-scale video data, to generate continuous HDR virtual light probe video frames containing reflection information from the surrounding scene within the masked region. Because the model operates directly in the HDR latent space, the generated reflections not only geometrically match the perspective of the surrounding scene but also conform to physical realism in terms of brightness and dynamic range, effectively solving the temporal flicker and illumination intensity truncation problems present in traditional methods.

[0064] (4) Environment Map Extraction and Projection Module

[0065] This module is responsible for converting the generated intermediate results into a common environment map format. The system extracts the image content of the virtual light probe region from the generated high dynamic range video frames. Subsequently, using a geometric projection transformation algorithm, the reflection information of the spherical surface is back-projected into an equidistant cylindrical projection. This process is performed frame by frame, and finally outputs a set of temporally coherent and physically realistic dynamic HDR environment map sequences, which can be directly used for virtual object rendering, scene relighting, and other applications in augmented reality. It supports robust estimation in fast camera movement and complex occlusion scenes.

[0066] (5) V-LITESET synthetic dataset

[0067] To enable the model to effectively learn dynamic HDR lighting, this system constructs a large-scale synthetic dataset called V-LITESET. This dataset is generated using a 3D rendering engine, utilizing panoramic video as a dynamic environment texture to wrap virtual scenes. The system places a virtual camera and an ideal reflective sphere at the center of the scene, simulating real-world physical camera parameters and rendering pipelines. After rendering, random exposure enhancements are applied to generate pairs of "LDR background videos" and "ground truth videos containing HDR reflective spheres." This dataset covers various complex indoor and outdoor lighting environments and dynamic changes, providing the model with precisely aligned and physically defined supervision signals, ensuring that the model can learn the mapping relationship from LDR backgrounds to infer HDR lighting.

[0068] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method.

[0069] Although embodiments and drawings of the present invention have been disclosed for illustrative purposes, those skilled in the art will understand that various substitutions, variations and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the scope of the present invention is not limited to the contents disclosed in the embodiments and drawings.

Claims

1. A high dynamic range illumination estimation method, characterized in that: The steps of the method are as follows: S1. Obtain the low dynamic range video sequence to be processed; S2. Construct video in-view task input, including applying a mask to the low dynamic range to occlude a specific area, and generating text prompts to describe the virtual light probe; S3. Using a pre-trained and finely tuned video diffusion model, based on the video in-painting task input, a virtual light probe video frame containing reflection information of the surrounding scene is generated in the occluded area; the video diffusion model directly outputs HDR latent features by learning a high dynamic range latent spatial representation. S4. Use an HDR-sensing decoder to reconstruct the HDR latent features into HDR video frames; S5. Extract reflection information from the virtual light probe region of the HDR video frame and generate a time-consistent dynamic HDR environment map sequence through projection transformation; The construction and training of the HDR latent space representation and HDR perceptual decoder in S3 and S4 are as follows: S3.1 Construct a variational autoencoder containing an encoder and a decoder; S3.2 Introduce a learnable tone mapping adapter and an inverse tone mapping adapter; the tone mapping adapter is used to convert the input linear HDR video frame into a logarithmic domain representation, and then input it into the encoder to generate latent features; S3.3 The decoder is used to decode the latent features into logarithmic domain features, and then transform them back into the linear HDR domain by the inverse tone mapping adapter; S3.

4. A phased fine-tuning strategy is adopted: In the first phase, the VAE parameters are frozen and only the tone mapping adapter is trained; in the second phase, the decoder and adapter are fine-tuned together to adapt the model to the HDR data distribution. The training process of the S3 video diffusion model adopts a low-rank adaptive fine-tuning strategy. The video diffusion model is fine-tuned using a pairwise dataset containing LDR videos, corresponding HDR ground truth videos, and text descriptions, so that it aligns with the distribution of the HDR latent space while maintaining the original LDR video generation prior. The specific area in S2 is the central area of ​​the video image, the text prompt describes a sphere with perfect specular reflection properties, and the projection transformation in S5 is to unfold the spherical reflection into an equidistant cylindrical projection.

2. The high dynamic range illumination estimation method according to claim 1, characterized in that: The calculation formula for the tone mapping adapter includes performing a logarithmic transformation on the input signal and adjusting it through convolutional layers, a learnable scaling factor, and a bias term to preserve illumination cues.

3. The high dynamic range illumination estimation method according to claim 1, characterized in that: The system is trained using a synthetic dataset containing dynamic lighting variations, which is generated by rendering virtual scenes in a 3D rendering engine using panoramic video as dynamic environment textures and applying random exposure enhancements.

4. A high dynamic range illumination estimation device, characterized in that, For performing the estimation method as described in any one of claims 1 to 3, comprising: The data acquisition module is used to acquire the LDR video sequence to be processed; The task building module is used to generate a masked video input and a text prompt describing the virtual light probe; The generation module contains a video diffusion model with an integrated tone mapping adapter for generating HDR format virtual light probe video content in the masked area. The post-processing module is used to extract and expand reflection information from the generated content and output a dynamic HDR environment map.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 3.

Images