Illumination perception video generation method and device based on renderer proxy reasoning, and storage medium

Abstract

The invention relates to an illumination perception video generation method and device based on renderer proxy reasoning and a storage medium. A user is supported to perform accurate and decoupling control on geometric layout, illumination conditions and camera tracks in a video generation process through a natural language instruction. A renderer agent is introduced to convert a text into a structured three-dimensional scene parameter, and a rendering engine is utilized to generate a two-dimensional scene agent comprising a diffuse reflection, gloss and roughness multi-channel layer; then, through a lightweight proxy encoder and an adapter, the physical illumination attribute is used as a strong condition signal to be injected into the video diffusion model. According to the method, the end-to-end generation from the text to the physical consistency video is realized, the generated video has accurate shadow, reflection and ambient light shielding effects while keeping realistic visual textures, the control precision and the automation degree of scene physical attributes are improved, and the method is suitable for the fields of film and television rehearsal, game asset production, virtual film production and the like.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and computer vision technology, specifically relating to a method, apparatus and storage medium for generating illumination-aware video based on renderer proxy inference. Background Technology

[0002] In recent years, with the development and popularization of artificial intelligence, video generation technology based on diffusion models has made significant progress. However, existing data-driven models still face significant challenges in terms of controllability. Specifically, existing video generation models often struggle to decouple and control the physical properties of a scene. For example, users find it difficult to precisely control the direction of lighting, shadow casting, material reflection properties, and the precise layout of objects in a scene. Although some work has attempted to enhance control by introducing 3D bounding boxes or camera trajectories, these methods mostly ignore the core physical element of "lighting." This often results in inconsistent lighting and shadows in the generated videos (such as incorrect highlight positions and missing shadows), making it difficult to meet the high requirements for physical realism in professional fields such as film production and virtual production. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method, apparatus and storage medium for generating light-aware video based on renderer proxy inference. By introducing "renderer proxy" and "scene proxy", explicit 3D physical constraints are injected into the video generation process to generate videos with high physical realism.

[0004] The technical problem solved by this invention is achieved through the following technical solution: A method for generating light-aware video based on renderer proxy inference, the steps of which are as follows: S1. Obtain the text description command input by the user; S2. Use the renderer agent to reason about the text description instructions and construct a rough 3D scene representation including geometric layout, lighting conditions and camera trajectory; S3. Using a 3D rendering engine, perform layered rendering based on the rough 3D scene representation to generate a 2D scene proxy composed of multi-channel rendering layers. The 2D scene proxy contains physical lighting attribute information. S4. Input the two-dimensional scene agent into the pre-trained video diffusion model as a conditional guidance signal; S5. Generate a final video using the video diffusion model. The final video retains the physical lighting and geometric structure defined by the two-dimensional scene proxy while possessing realistic visual textures.

[0005] Moreover, S2 specifically refers to: S2.1. Use a large language model to parse the text description instructions and construct a scene graph, wherein the scene graph defines object categories and their spatial relationships; S2.2. Retrieve the corresponding 3D model assets from the preset 3D asset library according to the scene map, and arrange them according to spatial relationships to generate the scene geometry. S2.3. Parse the lighting atmosphere description in the text description instruction, and retrieve or generate a matching high dynamic range (HDR) environment map as the lighting conditions; S2.4 Parse the camera movement description in the text description instruction and generate a camera pose sequence that changes over time as the camera trajectory.

[0006] Furthermore, the two-dimensional scene proxy of S3 is a stack consisting of a set of temporally continuous image sequences. The multi-channel rendering layer includes a diffuse pass, a glossy GGX pass, and a rough GGX pass. The diffuse pass is used to capture low-frequency ambient lighting information; the glossy GGX pass is used to capture high-frequency specular reflection information under low roughness; and the rough GGX pass is used to capture mid-frequency reflection information under high roughness.

[0007] Moreover, S4 specifically refers to: A lightweight proxy encoder is used to downsample and extract features from the two-dimensional scene proxy to generate scene proxy features; The scene proxy features are injected into the intermediate layer of the video diffusion model using a proxy adapter; The proxy adapter calculates the residual between the scene proxy features and the intermediate features of the video diffusion model, and then superimposes it onto the original video features via a learnable zero-initialization neural network.

[0008] Furthermore, the video diffusion model is trained using a three-stage progressive training strategy: a) Freeze the backbone parameters of the video diffusion model and train only the proxy encoder and proxy adapter, so that the model learns to translate scene proxies into control signals; b) Unfreeze the low-rank adaptation (LoRA) layer in the video diffusion model and fine-tune it together with the proxy encoder and adapter to balance controllability and generation quality; c) Mix real-world video data with synthetic video data in the training data and fine-tune the model together to enhance the model's ability to generalize to diverse lighting phenomena.

[0009] Furthermore, the synthesized video data is constructed in the following manner: a) Select 3D assets with physical properties and materials to construct a virtual scene; b) Use diverse HDR environment maps to illuminate the virtual scene and apply time-varying rotation operations to the HDR environment maps to simulate dynamic lighting. c) Render a composite video with dynamic lighting changes and its corresponding 2D scene proxy by moving the camera's perspective.

[0010] A light-aware video generation device based on renderer proxy inference includes: The instruction acquisition module is used to acquire text-described instructions; The proxy reasoning module is used to construct a rough 3D scene containing geometric, lighting, and camera parameters through semantic analysis; The proxy rendering module is used to generate a sequence of 2D scene proxy images containing diffuse, gloss, and roughness reflection channels based on the physical rendering engine. The video generation module includes a proxy encoder, a proxy adapter, and a video diffusion backbone network, used to generate physically consistent video content based on the two-dimensional scene.

[0011] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method.

[0012] The advantages and beneficial effects of this invention are as follows: 1. The intelligent agent module of this invention can convert users' natural language descriptions into structured 3D scene control signals. The agent first parses the text to construct a scene graph, retrieves 3D assets, and lays them out; secondly, it selects HDR environment maps based on the description; and finally, it plans the camera motion trajectory.

[0013] 2. This invention connects the 3D engine and the 2D video generation model by not directly using complex 3D mesh data, but instead utilizing a physical renderer to generate a set of "scene proxies." These proxies are a stacked set of 2D rendering channels, specifically including diffuse, glossy GGX, and rough GGX channels. This representation preserves the scene's geometric layout information while explicitly encoding the physical interaction information between lighting and materials (such as shading, reflection, and occlusion shadows).

[0014] 3. This invention constructs a video diffusion model based on Transformer and designs a lightweight "proxy encoder" and "proxy adapter". The encoder extracts the features of the scene proxy, and the adapter injects physical lighting cues into the latent space of video generation through residual connections, guiding the model to generate videos with realistic textures and conforming to the laws of physical lighting.

[0015] 4. To prevent strong physical constraints from damaging the generative ability of the pre-trained model, this invention adopts a three-stage training approach: first, train the adapter; then, introduce LoRA for joint fine-tuning; and finally, mix real and synthetic data to enhance diversity.

[0016] 5. By introducing a scene proxy containing physical information, the present invention generates videos with accurate reflection, shadow and ambient light occlusion effects.

[0017] 6. Users of this invention can achieve independent control over video lighting, layout, and camera by modifying the 3D proxy of the intermediate layer (such as changing HDR textures, moving objects, or moving the camera).

[0018] 7. The renderer agent of this invention realizes automatic conversion from text to 3D control signals, reducing the threshold for users to create complex 3D scenes. Attached Figure Description

[0019] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0020] The present invention will be further described in detail below through specific embodiments. The following embodiments are merely descriptive and not limiting, and should not be used to limit the scope of protection of the present invention.

[0021] This invention proposes a light-aware video generation method based on renderer proxy inference, which is used to achieve precise and decoupled control of geometric layout, lighting conditions and camera trajectory during video generation. The method takes user natural language description instructions as input and outputs video content with both physical consistency and realistic visual texture.

[0022] To achieve the above functions, this invention constructs a video generation framework (LiVER) that integrates large language model inference and a physically based rendering engine. Its core innovation lies in introducing a "renderer proxy" to transform text into structured 3D scene parameters, and utilizing a "2D scene proxy" as a strong physical condition signal injected into the video diffusion model. This invention also includes a three-stage progressive training strategy that balances physical constraints and generation quality by mixing real-world video with synthetic video data constructed based on physically based rendering. The overall technical solution is as follows: Figure 1 As shown.

[0023] An innovative method for generating light-aware video based on renderer proxy inference, characterized by the following steps: S1. Obtain the text description command input by the user; S2. Use the renderer agent to reason about the text description instructions and construct a rough 3D scene representation including geometric layout, lighting conditions and camera trajectory; S2.1. Use a large language model to parse the text description instructions and construct a scene graph, wherein the scene graph defines object categories and their spatial relationships; S2.2. Retrieve the corresponding 3D model assets from the preset 3D asset library according to the scene map, and arrange them according to spatial relationships to generate the scene geometry. S2.3. Parse the lighting atmosphere description in the text description instruction, and retrieve or generate a matching high dynamic range (HDR) environment map as the lighting conditions; S2.4 Parse the camera movement description in the text description instruction and generate a camera pose sequence that changes over time as the camera trajectory.

[0024] S3. Using a 3D rendering engine, perform layered rendering based on the rough 3D scene representation to generate a 2D scene proxy composed of multi-channel rendering layers. The 2D scene proxy contains physical lighting attribute information. The 2D scene proxy is a stack composed of a set of temporally continuous image sequences. The multi-channel rendering layers include a diffuse pass, a glossy GGX pass, and a rough GGX pass. The diffuse pass is used to capture low-frequency ambient lighting information; the glossy GGX pass is used to capture high-frequency specular reflection information under low roughness; and the rough GGX pass is used to capture mid-frequency reflection information under high roughness.

[0025] S4. Input the two-dimensional scene agent into the pre-trained video diffusion model as a conditional guidance signal; A lightweight proxy encoder is used to downsample and extract features from the two-dimensional scene proxy to generate scene proxy features; The scene proxy features are injected into the intermediate layer of the video diffusion model using a proxy adapter; The proxy adapter calculates the residual between the scene proxy features and the intermediate features of the video diffusion model, and then superimposes it onto the original video features via a learnable zero-initialization neural network.

[0026] S5. Generate a final video using the video diffusion model. The final video retains the physical lighting and geometric structure defined by the two-dimensional scene proxy while possessing realistic visual textures.

[0027] The video diffusion model is trained using a three-stage progressive training strategy: a) Freeze the backbone parameters of the video diffusion model and train only the proxy encoder and proxy adapter, so that the model learns to translate scene proxies into control signals; b) Unfreeze the low-rank adaptation (LoRA) layer in the video diffusion model and fine-tune it together with the proxy encoder and adapter to balance controllability and generation quality; c) Mix real-world video data with synthetic video data in the training data and fine-tune the model together to enhance the model's ability to generalize to diverse lighting phenomena.

[0028] Furthermore, the synthesized video data is constructed in the following manner: a) Select 3D assets with physical properties and materials to construct a virtual scene; b) Use diverse HDR environment maps to illuminate the virtual scene and apply time-varying rotation operations to the HDR environment maps to simulate dynamic lighting. c) Render a composite video with dynamic lighting changes and its corresponding 2D scene proxy by moving the camera's perspective.

[0029] A light-aware video generation device based on renderer proxy inference includes: (1) Renderer proxy inference module First, the system acquires the user's natural language description instructions. Then, a large language model acts as an intelligent agent to perform semantic parsing and reasoning on the instructions. This agent first constructs a scene graph defining object categories and their spatial topological relationships; based on this scene graph, it retrieves corresponding model assets from a pre-built 3D asset library and completes the geometric layout. Simultaneously, the agent parses the lighting atmosphere description in the instructions, retrieves or generates matching high dynamic range (HDR) environment maps as lighting conditions, and parses the camera movement description to generate a time-varying camera pose sequence. Through these steps, the system constructs a coarse 3D scene representation containing geometric, lighting, and camera parameters, achieving automatic conversion from textual semantics to structured 3D parameters.

[0030] (2) Two-dimensional scene proxy generation module This module is responsible for connecting the 3D physical world with the 2D generated model. Utilizing a 3D rendering engine, it performs layered rendering based on the aforementioned rough 3D scene representation, generating a stack of temporally continuous image sequences, known as a "2D scene proxy." Unlike directly using a 3D mesh, this proxy contains explicit physically based lighting properties, specifically composed of multi-channel rendering layers: a Diffuse Pass for capturing low-frequency ambient lighting information; a Glossy GGX Pass for capturing high-frequency specular reflection information under low roughness; and a Rough GGX Pass for capturing mid-frequency reflection information under high roughness. These channels preserve the scene's geometric layout and encode physical interaction cues between lighting and materials.

[0031] (3) Physical condition injection and video generation module This module is responsible for introducing a 2D scene proxy as a strong conditional signal into a pre-trained video diffusion model. First, a lightweight proxy encoder is used to downsample and extract features from the 2D scene proxy, generating scene proxy features. Then, a proxy adapter injects these scene proxy features into the intermediate layer of the video diffusion model. The proxy adapter calculates the residual between the scene proxy features and the intermediate features of the video diffusion model, and then superimposes it onto the original video features via a learnable, zero-initialization neural network. This mechanism ensures that the final generated video retains the physical lighting (such as accurate shadows and reflections) and geometry defined by the 2D scene proxy while possessing realistic visual texture.

[0032] (4) Synthetic video data construction module To enhance the model's understanding and generalization ability regarding physically based lighting phenomena, this system constructs a synthetic video dataset based on physically based rendering. Specifically, a virtual scene is constructed using 3D assets with physically based materials; diverse HDR environment maps are used to illuminate the virtual scene, and time-varying rotation operations are applied to the HDR environment maps to simulate dynamic lighting; finally, rendering is performed by moving the camera's viewpoint to obtain a synthetic video with dynamic lighting changes and its corresponding 2D scene proxy. This data construction method provides the model with physically accurate "ground truth" supervision, compensating for the difficulty in decoupling physical properties in real video data.

[0033] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method.

[0034] Although embodiments and drawings of the present invention have been disclosed for illustrative purposes, those skilled in the art will understand that various substitutions, variations and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the scope of the present invention is not limited to the contents disclosed in the embodiments and drawings.

Claims

1. A method for generating light-aware video based on renderer proxy inference, characterized in that: The steps of the method are as follows: S1. Obtain the text description command input by the user; S2. Use the renderer agent to reason about the text description instructions and construct a rough 3D scene representation including geometric layout, lighting conditions and camera trajectory; S3. Using a 3D rendering engine, perform layered rendering based on the rough 3D scene representation to generate a 2D scene proxy composed of multi-channel rendering layers. The 2D scene proxy contains physical lighting attribute information. S4. Input the two-dimensional scene agent into the pre-trained video diffusion model as a conditional guidance signal; S5. Generate a final video using the video diffusion model. The final video retains the physical lighting and geometric structure defined by the two-dimensional scene proxy while possessing realistic visual textures.

2. The method for generating light-aware video based on renderer proxy inference according to claim 1, characterized in that: Specifically, S2 is: S2.

1. Use a large language model to parse the text description instructions and construct a scene graph, wherein the scene graph defines object categories and their spatial relationships; S2.

2. Retrieve the corresponding 3D model assets from the preset 3D asset library according to the scene map, and arrange them according to spatial relationships to generate the scene geometry. S2.

3. Parse the lighting atmosphere description in the text description instruction, and retrieve or generate a matching high dynamic range (HDR) environment map as the lighting conditions; S2.4 Parse the camera movement description in the text description instruction and generate a camera pose sequence that changes over time as the camera trajectory.

3. The method for generating light-aware video based on renderer proxy inference according to claim 1, characterized in that: The two-dimensional scene proxy of S3 is a stack consisting of a set of temporally continuous image sequences. The multi-channel rendering layer includes a diffuse pass, a glossy GGX pass, and a rough GGX pass. The diffuse pass is used to capture low-frequency ambient lighting information. The Glossy GGX Pass is used to capture high-frequency specular reflection information under low roughness conditions; the Rough GGX Pass is used to capture mid-frequency reflection information under high roughness conditions.

4. The method for generating light-aware video based on renderer proxy inference according to claim 1, characterized in that: Specifically, S4 is: A lightweight proxy encoder is used to downsample and extract features from the two-dimensional scene proxy to generate scene proxy features; The scene proxy features are injected into the intermediate layer of the video diffusion model using a proxy adapter; The proxy adapter calculates the residual between the scene proxy features and the intermediate features of the video diffusion model, and then superimposes it onto the original video features via a learnable zero-initialization neural network.

5. The method for generating light-aware video based on renderer proxy inference according to claim 1, characterized in that: The video diffusion model is trained using a three-stage progressive training strategy: a) Freeze the backbone parameters of the video diffusion model and train only the proxy encoder and proxy adapter, so that the model learns to translate scene proxies into control signals; b) Unfreeze the low-rank adaptation (LoRA) layer in the video diffusion model and fine-tune it together with the proxy encoder and adapter to balance controllability and generation quality; c) Mix real-world video data with synthetic video data in the training data and fine-tune the model together to enhance the model's ability to generalize to diverse lighting phenomena.

6. The method for generating light-aware video based on renderer proxy inference according to claim 5, characterized in that: The synthesized video data is constructed in the following manner: a) Select 3D assets with physical properties and materials to construct a virtual scene; b) Use diverse HDR environment maps to illuminate the virtual scene and apply time-varying rotation operations to the HDR environment maps to simulate dynamic lighting. c) Render a composite video with dynamic lighting changes and its corresponding 2D scene proxy by moving the camera's perspective.

7. A light-aware video generation device based on renderer proxy inference, characterized in that, include: The instruction acquisition module is used to acquire text-described instructions; The proxy reasoning module is used to construct a rough 3D scene containing geometric, lighting, and camera parameters through semantic analysis; The proxy rendering module is used to generate a sequence of 2D scene proxy images containing diffuse, gloss, and roughness reflection channels based on the physical rendering engine. The video generation module includes a proxy encoder, a proxy adapter, and a video diffusion backbone network, used to generate physically consistent video content based on the two-dimensional scene.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 6.

Images