High dynamic range illumination estimation method and device and storage medium

Abstract

The invention relates to a high-dynamic-range illumination estimation method and device and a storage medium, and aims to recover a physically real high-dynamic-range environment map with consistent time sequence from a common low-dynamic-range video sequence by utilizing space-time generation prior of a video generation model and constructing a video interior drawing task. Applying a mask to an input video, and guiding the model to generate a virtual optical probe containing surrounding environment reflection information in the center of a picture in combination with a specific text prompt; a high dynamic range perception variational auto-encoder is constructed, and a learnable tone mapping and inverse tone mapping adapter is introduced to enable a video diffusion model to directly understand and output high dynamic range illumination features in a logarithm domain submerged space. According to the method, a high-quality dynamic high-dynamic range environment sequence can be output, higher robustness is achieved when dynamic scenes, rapid camera motion and complex shielding are processed, and the method is suitable for application such as augmented reality, virtual object implantation and scene re-illumination.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision and graphics, and particularly relates to a high dynamic range light estimation method and device and a storage medium. BACKGROUND

[0002] Light estimation, as a key method for realizing virtual-real fusion in virtual content creation, has become an important direction of research in the fields of virtual reality (VR) and augmented reality (AR). Accurate light estimation is the basis for realizing realistic augmented reality, virtual object insertion and scene relighting.

[0003] Traditional physical-based methods usually assume that light is static or simplified, making it difficult to handle complex dynamic scenes. Existing deep learning-based methods mostly focus on light estimation for a single image, which leads to a lack of temporal consistency in light estimation between frames when processing videos, resulting in flickering.

[0004] In addition, although existing video generation models can generate realistic light and shadow effects, their internal representations are usually optimized for low dynamic range (LDR) content and cannot directly output physically realistic high dynamic range (HDR) light information. Simply using a video model as an LDR to HDR translator often ignores the inherent generation priors of the model itself. SUMMARY

[0005] The purpose of the present application is to overcome the shortcomings of the prior art and provide a high dynamic range light estimation method, device and storage medium. The inherent light understanding ability of the video generation model is utilized to realize light estimation through a reconstruction task form. The light estimation task is reconstructed as a "video inpainting" task, that is, a virtual "light probe" is inserted into the input video, forcing the video generation model to generate physically plausible reflections on the probe surface according to the surrounding spatio-temporal context, and then inferring the ambient light.

[0006] The technical problem of the present application is solved by the following technical solution: A high dynamic range light estimation method, the steps of the method are: S1, obtaining a low dynamic range (LDR) video sequence to be processed; S2, constructing a video inpainting task input, including applying a mask to the low dynamic range to block certain areas, and generating a text prompt word for describing a virtual light probe; S3, generating a virtual light probe video frame containing surrounding scene reflection information in the occluded region based on the video-in-painting task input, by using a pre-trained and fine-tuned video diffusion model; the video diffusion model directly outputs an HDR latent feature by learning a high dynamic range (HDR) latent space representation; S4, reconstructing the HDR latent feature into an HDR video frame by using an HDR perceptual decoder; S5, extracting reflection information from the virtual light probe region of the HDR video frame and generating a time-consistent dynamic HDR environment map sequence through projection transformation.

[0007] Moreover, the construction and training of the HDR latent space representation and the HDR perceptual decoder in S3 and S4 are as follows: S2.1, constructing a variational autoencoder (VAE) containing an encoder and a decoder; S2.2, introducing a learnable tone mapping adapter and an inverse tone mapping adapter; the tone mapping adapter is used to convert the input linear HDR video frame into a logarithmic domain representation, and then input the encoder to generate a latent feature; S2.3, the decoder is used to decode the latent feature into a logarithmic domain feature, and then transform it back to the linear HDR domain through the inverse tone mapping adapter; S2.4, adopting a phased fine-tuning strategy: in the first phase, freeze the VAE parameters and only train the tone mapping adapter; in the second phase, fine-tune the decoder and the adapter jointly to adapt the model to the HDR data distribution.

[0008] Moreover, the calculation formula of the tone mapping adapter includes logarithmic transformation of the input signal, and adjustment through convolution layer, learnable scaling factor and bias term to preserve the illumination clues.

[0009] Moreover, the training process of the video diffusion model in S3 adopts a low rank adaptation (LoRA) fine-tuning strategy; a paired data set containing LDR video, corresponding HDR ground truth video (containing virtual light probe) and text description is used to fine-tune the video diffusion model, so that it can align the distribution of the HDR latent space while maintaining the original LDR video generation prior.

[0010] Moreover, the specific region in S2 is the center region of the video picture, and the text prompt word describes a sphere with perfect specular reflection properties, and the projection transformation in S5 is to expand the spherical reflection into an equidistant cylindrical projection.

[0011] Moreover, a synthetic data set containing dynamic light changes is used for training, which is generated by rendering a virtual scene in a three-dimensional rendering engine using a panoramic video as a dynamic environment texture and applying random exposure enhancement.

[0012] A high dynamic range illumination estimation device, comprising: A data acquisition module for acquiring an LDR video sequence to be processed; A task construction module for generating a video input with a mask and a text prompt describing a virtual light probe; A generation module comprising a video diffusion model integrated with a tone mapping adapter for generating virtual light probe video content in HDR format in the mask area; A post-processing module for extracting and unfolding reflection information from the generated content and outputting a dynamic HDR environment map.

[0013] A computer readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the method.

[0014] The advantages and beneficial effects of the present application are: 1. The present application designs a VAE (Variational Autoencoder) for HDR perception, which maps the input linear HDR video to a logarithmic domain latent space by introducing a learnable tone mapping adapter, ensuring that the model can handle extreme changes in illumination intensity without losing details.

[0015] 2. The present application uses a pre-trained video diffusion Transformer model to generate a chrome ball in the occluded area of the LDR video, combining a text prompt describing a "perfect mirror reflection sphere".

[0016] 3. The present application uses LoRA (Low Rank Adaptation) technology to fine-tune the video diffusion model, making it adapt to the distribution of the HDR latent space while retaining the spatiotemporal consistency prior learned by the pre-trained model on large-scale video data.

[0017] 4. The present application extracts the generated chrome ball region from the HDR video frame and generates an HDR environment map in an equirectangular projection through back projection.

[0018] 5. The present application constructs a large-scale synthetic dataset (V-LITESET) containing video pairs with dynamic lighting changes generated from panoramic videos for model training.

[0019] 6. The present application uses the spatiotemporal attention mechanism of the video generation model, and the generated environment map is highly coherent in time, avoiding the flickering problem of single-frame methods.

[0020] 7. The present application uses an improved HDR VAE, which can directly infer in the HDR domain and output physically realistic luminance information, rather than pseudo-HDR.

[0021] 8、The present application converts illumination estimation into a generation task, and can handle extreme cases such as fast camera motion and complex occlusion. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 A flowchart of the present application. DETAILED DESCRIPTION

[0023] The present application will be further described in detail below through specific examples, which are only descriptive and not limiting, and cannot limit the protection scope of the present application.

[0024] The present application proposes a dynamic high dynamic range (HDR) illumination estimation method based on a video generation model, which is used to recover a time-consistent and physically realistic high dynamic range environment map from a normal low dynamic range (LDR) video sequence. The method takes monocular LDR video as input, constructs an "in-video rendering" task, combines specific text prompts, uses the spatio-temporal prior of the video generation model to infer the environment illumination, and outputs a high-quality dynamic HDR environment map sequence.

[0025] To achieve the above functions, the present application constructs a HDR-aware variational autoencoder (VAE) and a video diffusion model architecture with adaptive fine-tuning, the core of which is to introduce a learnable tone mapping adapter to process HDR information in the logarithmic domain latent space, and to maintain the spatio-temporal consistency of the generation model through a low-rank adaptation (LoRA) strategy. The present application also provides a synthetic dataset V-LITESET for training the model, which contains video pairs with rich dynamic illumination changes, improving the robustness of the model in complex scenes. The overall technical solution is as shown in Figure 1 .

[0026] A high dynamic range illumination estimation method, the innovation of which is that the steps of the method are: S1, obtaining a low dynamic range (LDR) video sequence to be processed; S2, constructing an in-video rendering task input, including applying a mask to the low dynamic range to occlude specific areas, and generating text prompts for describing virtual light probes; S3, using a pre-trained and fine-tuned video diffusion model to generate virtual light probe video frames containing surrounding scene reflection information in the occluded area based on the in-video rendering task input; the video diffusion model directly outputs HDR latent features by learning HDR latent space representation; S4, using an HDR-aware decoder to reconstruct the HDR latent features into HDR video frames; S5, extracting reflection information from the virtual light probe area of the HDR video frames, and generating a time-consistent dynamic HDR environment map sequence through projection transformation.

[0027] The construction and training of the HDR latent spatial representation and HDR perceptual decoder are as follows: S2.1 Construct a variational autoencoder (VAE) containing an encoder and a decoder. S2.2 Introduce a learnable tone mapping adapter and an inverse tone mapping adapter; the tone mapping adapter is used to convert the input linear HDR video frame into a logarithmic domain representation, and then input it into the encoder to generate latent features; S2.3 The decoder is used to decode the latent features into logarithmic domain features, and then transform them back into the linear HDR domain by the inverse tone mapping adapter; S2.4. A phased fine-tuning strategy is adopted: In the first phase, the VAE parameters are frozen and only the tone mapping adapter is trained; in the second phase, the decoder and adapter are fine-tuned together to adapt the model to the HDR data distribution.

[0028] The calculation formula for the tone mapping adapter includes performing a logarithmic transformation on the input signal and adjusting it through convolutional layers, a learnable scaling factor, and a bias term to preserve illumination cues.

[0029] The training process of the S3 video diffusion model adopts a low-rank adaptation (LoRA) fine-tuning strategy. The video diffusion model is fine-tuned using a pairwise dataset containing LDR videos, corresponding HDR ground truth videos (including virtual light probes), and text descriptions, so that it aligns with the distribution of the HDR latent space while maintaining the original LDR video generation prior.

[0030] The specific area in S2 is the central area of ​​the video image, the text prompt describes a sphere with perfect specular reflection properties, and the projection transformation in S5 is to unfold the spherical reflection into an equidistant cylindrical projection.

[0031] This invention utilizes a synthetic dataset containing dynamic lighting changes for training. This dataset is generated by rendering virtual scenes in a 3D rendering engine using panoramic video as a dynamic environment texture and applying random exposure enhancement.

[0032] A high dynamic range illumination estimation device, its innovation lies in: including: (1) Video in-video drawing task construction module: First, a low dynamic range (LDR) video sequence is acquired as the basic input. To infer ambient lighting using the generative model, a video in-frame rendering task is constructed. Specifically, the system automatically applies a mask to the central region of the input video frame to occlude it, while simultaneously generating a text prompt describing a "perfectly specular reflective sphere." This step aims to force the video generation model to "draw" a virtual light probe that reflects the surrounding environment in the occluded area, based on the spatiotemporal context information of the unoccluded area and utilizing its internally learned physical laws. This approach transforms the lighting estimation problem into a video completion problem, fully leveraging the video model's understanding of scene geometry and lighting relationships.

[0033] (2) HDR-Perceptual Variational Autoencoder (HDR-VAE): This module addresses the limitation of traditional video generation models, which can only process LDR data. The system constructs a variational autoencoder (VAE) comprising an encoder and a decoder, and introduces a learnable tone mapping adapter and an inverse tone mapping adapter. The tone mapping adapter, located before the encoder, uses logarithmic transformation combined with convolutional layers, a learnable scaling factor, and a bias term to convert the input linear HDR video frames into a logarithmic domain suitable for model processing, thereby compressing the dynamic range while preserving physical cues in the highlights. The inverse tone mapping adapter, located after the decoder, is responsible for accurately restoring the decoded logarithmic domain features back to the linear HDR domain.

[0034] This module enables the video diffusion model to perform feature learning and inference directly in the HDR latent space, rather than simply outputting pixel colors, thus establishing an accurate representation of physical illumination information.

[0035] (3) Video Diffusion Generation Network Based on Stream Matching This module utilizes a pre-trained stream-matching video diffusion Transformer model to perform generation tasks within the latent space constructed by HDR-VAE. Inputs include latent features of the masked LDR video and text cue word features. The model leverages its spatiotemporal attention mechanism, pre-trained on large-scale video data, to generate continuous HDR virtual light probe video frames containing reflection information from the surrounding scene within the masked region. Because the model operates directly in the HDR latent space, the generated reflections not only geometrically match the perspective of the surrounding scene but also conform to physical realism in terms of brightness and dynamic range, effectively solving the temporal flicker and illumination intensity truncation problems present in traditional methods.

[0036] (4) Environment Map Extraction and Projection Module This module is responsible for converting the generated intermediate results into a common environment map format. The system extracts the image content of the virtual light probe region from the generated high dynamic range video frames. Subsequently, using a geometric projection transformation algorithm, the reflection information of the spherical surface is back-projected into an equidistant cylindrical projection. This process is performed frame by frame, and finally outputs a set of temporally coherent and physically realistic dynamic HDR environment map sequences, which can be directly used for virtual object rendering, scene relighting, and other applications in augmented reality. It supports robust estimation in fast camera movement and complex occlusion scenes.

[0037] (5) V-LITESET synthetic dataset To enable the model to effectively learn dynamic HDR lighting, this system constructs a large-scale synthetic dataset called V-LITESET. This dataset is generated using a 3D rendering engine, utilizing panoramic video as a dynamic environment texture to wrap virtual scenes. The system places a virtual camera and an ideal reflective sphere at the center of the scene, simulating real-world physical camera parameters and rendering pipelines. After rendering, random exposure enhancements are applied to generate pairs of "LDR background videos" and "ground truth videos containing HDR reflective spheres." This dataset covers various complex indoor and outdoor lighting environments and dynamic changes, providing the model with precisely aligned and physically defined supervision signals, ensuring that the model can learn the mapping relationship from LDR backgrounds to infer HDR lighting.

[0038] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method.

[0039] Although embodiments and drawings of the present invention have been disclosed for illustrative purposes, those skilled in the art will understand that various substitutions, variations and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the scope of the present invention is not limited to the contents disclosed in the embodiments and drawings.

Claims

1. A high dynamic range illumination estimation method, characterized in that: The steps of the method are: S1, obtaining a low dynamic range video sequence to be processed; S2, constructing a video in-drawing task input, including applying a mask to the low dynamic range to shield a specific area, and generating a text prompt word for describing a virtual light probe; S3, using a pre-trained and fine-tuned video diffusion model to generate a virtual light probe video frame containing surrounding scene reflection information in the shielded area based on the video in-drawing task input; the video diffusion model directly outputs an HDR latent feature by learning a high dynamic range latent space representation; S4, using an HDR perception decoder to reconstruct the HDR latent feature into an HDR video frame; S5, extracting reflection information from the virtual light probe area of the HDR video frame and generating a time-consistent dynamic HDR environment map sequence through projection transformation.

2. The high dynamic range illumination estimation method of claim 1, wherein: The construction and training of the HDR latent space representation and the HDR perception decoder in S3 and S4 are as follows: S3.1, constructing a variational autoencoder including an encoder and a decoder; S3.2, introducing a learnable tone mapping adapter and an inverse tone mapping adapter; the tone mapping adapter is used to convert the input linear HDR video frame into a logarithmic domain representation, and then input the encoder to generate a latent feature; S3.3, the decoder is used to decode the latent feature into a logarithmic domain feature, and then transform it back to the linear HDR domain through the inverse tone mapping adapter; S3.4, a phased fine-tuning strategy is adopted: in the first phase, the VAE parameters are frozen, and only the tone mapping adapter is trained; in the second phase, the decoder and the adapter are fine-tuned jointly, so that the model adapts to the HDR data distribution.

3. The high dynamic range illumination estimation method of claim 2, wherein: The calculation formula of the tone mapping adapter includes logarithmic transformation of the input signal, and adjustment through convolution layer, learnable scaling factor and bias term to preserve the illumination clues.

4. The high dynamic range illumination estimation method of claim 1, wherein: The training process of the video diffusion model in S3 adopts a low-rank adaptive fine-tuning strategy; a paired data set containing LDR video, corresponding HDR true value video and text description is used to fine-tune the video diffusion model, so that it aligns the distribution of the HDR latent space while maintaining the original LDR video generation prior.

5. The method of claim 1, wherein: The specific area in S2 is the center area of the video picture, and the text prompt word describes a sphere with perfect specular reflection properties; in S5, the projection transformation is to expand the spherical reflection into an equidistant cylindrical projection.

6. The high dynamic range illumination estimation method of claim 1, wherein: The training is performed using a synthetic data set containing dynamic light changes, which is generated by rendering a virtual scene in a three-dimensional rendering engine using a panoramic video as a dynamic environment texture and applying random exposure enhancement.

7. A high dynamic range illumination estimation apparatus, characterized by, It comprises: a data acquisition module for acquiring an LDR video sequence to be processed; a task construction module for generating a video input with a mask and a text prompt describing a virtual light probe; a generation module containing a video diffusion model integrated with a tone mapping adapter, for generating HDR format virtual light probe video content in the masked area; a post-processing module for extracting and expanding reflection information from the generated content and outputting a dynamic HDR environment map.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by a processor to implement the method of any one of claims 1-6.

Images