A global neural rendering method and system for multi-variable hybrid representation scenes

Abstract

This invention discloses a global neural rendering method and system for multi-variable hybrid scene representation, belonging to the field of computer graphics technology. It includes: a scene representation module for loading multi-variable hybrid scene data; a geometry module for generating a multi-channel geometry buffer based on the multi-variable hybrid scene data; a lighting module containing at least one neural network sub-module for performing lighting calculations based on the multi-channel geometry buffer; a scheduling and synchronization module for coordinating cross-module data flow through a rendering graph structure and performing resource merging and task scheduling between the geometry and lighting modules based on a unified request structure; and a merged output module for fusing neural network lighting output and traditional rendering output to generate the final frame image to complete global neural rendering. This invention integrates multi-variable scene data through modular design and utilizes a unified interface and scheduling structure to achieve efficient collaboration between traditional and neural rendering, improving image quality and system flexibility.

Description

Technical Field

[0001] This invention belongs to the field of computer graphics technology, specifically relating to a global neural rendering method and system for multi-variable hybrid representation scenes. Background Technology

[0002] In recent years, neural rendering, as a cutting-edge field at the intersection of computer graphics and computer vision, has received widespread attention. By introducing deep learning models, it attempts to bridge the gap between realistic images and traditional graphics rendering, achieving efficient modeling and realistic reproduction of complex geometry, materials, and lighting environments. Traditional rendering pipelines rely on explicit modeling, which, while flexible, incurs extremely high computational costs when handling highly complex scenes (such as detailed geometry and global illumination). Neural rendering, on the other hand, leverages the advantages of deep neural networks in learning representations and generating data, transforming the image generation problem into an end-to-end learning process, significantly improving image realism and maintaining cross-viewpoint consistency.

[0003] In modern graphics engines, deferred rendering decouples geometry from lighting, generating a G-Buffer that includes information such as normals, depth, and material parameters. Based on this structure, a series of neural network post-processing methods have been proposed. For example, using CNNs or Transformers to perform neural decoding on the G-Buffer enables lighting reconstruction, material enhancement, anti-aliasing, and noise reduction. Simultaneously, neural networks are also used to reconstruct missing subsurface scattering, reflective highlights, or translucent effects, overcoming the limitations of traditional rasterization in terms of realism. Furthermore, some methods model traditional BSDF (Bidirectional Spread Function) parameters as learnable modules, making material representation more expressive. In texture generation for complex scenes, neural networks can also be used for texture super-resolution, completion, and baking, thereby reducing reliance on art resources and improving content generation efficiency.

[0004] Gaussian sputtering is an emerging 3D reconstruction and rendering technique that models scenes or objects using an explicit representation of a large number of Gaussian distribution points. Compared to earlier Neural Radiation Field (NeRF) methods based on implicit volume, Gaussian sputtering achieves significant improvements in training speed, rendering efficiency, and image quality. Thanks to its explicit representation, this method can also serve as a new way to visualize scenes within traditional graphics rendering pipelines, possessing good integration potential and practical application value. With the development of deep learning, neural network rendering is gradually expanding from end-to-end image generation to deep integration with traditional graphics pipelines. Compared to fully data-driven volumetric neural rendering (such as NeRF), fusion neural rendering methods focus more on collaboration with existing real-time rendering systems (such as Unity and Unreal Engine), aiming to enhance image quality and intelligence by introducing neural networks while retaining the efficiency and flexibility of traditional rendering.

[0005] This type of fusion-based neural rendering method combines controllability and intelligence, making it suitable for applications such as real-time rendering, game art toolchains, virtual human-driven systems, and digital content creation. However, some problems still exist: there is a significant semantic gap between the input-output structure of neural networks and traditional graphics pipelines, making it difficult to build a unified end-to-end scheduling mechanism; currently, frameworks mostly exist in the form of plugins or post-processing, failing to integrate neural processing logic and rendering task scheduling at the pipeline level, thus hindering the full release of the potential of neural networks in scene understanding and contextual reasoning. Furthermore, there is a lack of complete mechanisms in resource scheduling, memory management, and real-time control, making it impossible to guarantee stable operation in multi-resolution and multi-tasking environments. Therefore, there is an urgent need for a new rendering framework that can deeply integrate neural processing logic into traditional graphics rendering pipelines, not only improving the realism of images but also possessing good scalability, resource controllability, and deployment flexibility to adapt to the ever-increasing demand for real-time graphics rendering and diverse application scenarios. Summary of the Invention

[0006] In view of the above, the purpose of this invention is to provide a global neural rendering method and system for multi-variable hybrid representation scenes. By adopting a modular scheduling structure and a unified interface, traditional explicit rendering and neural network rendering are combined to achieve deep integration of neural processing logic in the traditional graphics rendering pipeline. While ensuring real-time rendering efficiency, this significantly improves image quality and system scalability and flexibility, and is suitable for real-time graphics rendering needs and diverse global neural rendering application scenarios.

[0007] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0008] In a first aspect, embodiments of the present invention provide a global neural rendering system for a multi-variable hybrid representation scene, comprising:

[0009] The scene representation module is used to load diverse and mixed scene data;

[0010] The geometry module, which is connected to the scene representation module, is used to generate a multi-channel geometry buffer based on multi-source mixed scene data.

[0011] The lighting module is communicatively connected to the geometry module and contains at least one neural network submodule for performing lighting calculations based on a multi-channel geometry buffer.

[0012] The scheduling and synchronization module communicates with the geometry module and the lighting module, coordinates cross-module data flow through the rendering graph structure, and performs resource merging and task scheduling based on a unified request structure.

[0013] The merged output module is connected to the lighting module and is used to fuse the neural network lighting output with the traditional rendering output to generate the final frame image to complete the global neural rendering.

[0014] Preferably, the multivariate hybrid scene data includes traditional explicit representation and neural field representation.

[0015] Preferably, the multi-channel geometric buffer includes:

[0016] Based on traditional explicit representation using geometric information and material properties generated through methods including rasterization or ray tracing;

[0017] The neural field representation is based on intermediate rendering features generated using methods including projection sampling, Gaussian sputtering, or volumetric path sampling.

[0018] Preferably, the illumination module includes at least one of the following neural network sub-modules:

[0019] Lighting reconstruction subnetwork, material perception enhancement subnetwork, screen space effects subnetwork, dynamic resolution adaptation and super-resolution subnetwork.

[0020] Preferably, the scheduling and synchronization module performs the following operations:

[0021] Encapsulate the data requirements of submodules into a unified request structure;

[0022] Merge duplicate requests and overwrite low-resolution requests with high-resolution requests;

[0023] A unified rendering encoding for representing diverse and mixed scenes is achieved through type identifier bits.

[0024] Preferably, the geometry module adopts a modular shader template, and the output logic is dynamically configured through macro definitions. When adding a new scene representation type, only the differential shader code needs to be extended without refactoring the rendering pipeline.

[0025] Preferably, in the rendering graph structure, each rendering stage is regarded as a graph node, and the input and output resources of each graph node are connected to other nodes through boundary dependencies to form a directed acyclic graph.

[0026] Secondly, embodiments of the present invention also provide a global neural rendering method for multi-variable hybrid representation scenes, implemented using the aforementioned global neural rendering system for multi-variable hybrid representation scenes, comprising the following steps:

[0027] S1, using the scene expression module to load multi-dimensional mixed scene data;

[0028] S2 utilizes the geometry module to generate a multi-channel geometry buffer based on multi-variable mixed scene data;

[0029] S3, using the lighting module to perform lighting calculations based on a multi-channel geometric buffer through at least one neural network sub-module;

[0030] S4 utilizes the scheduling and synchronization module to coordinate cross-module data flow through a rendering graph structure, and performs resource merging and task scheduling based on a unified request structure.

[0031] S5 utilizes the merged output module to fuse neural network lighting output with traditional rendering output to generate the final frame image to complete global neural rendering.

[0032] Thirdly, embodiments of the present invention also provide an electronic device, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-described global neural rendering method for multi-variable hybrid representation scenes when executing the computer program.

[0033] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the aforementioned global neural rendering method for multi-variable hybrid representation scenes.

[0034] Compared with the prior art, the beneficial effects of the present invention include at least the following:

[0035] The global neural rendering method and system for multi-scene hybrid representation provided by this invention combines traditional explicit rendering with neural network technology. It significantly improves image quality, system flexibility, and deployment scalability while ensuring real-time rendering efficiency. By adopting a modular scheduling structure and a unified interface, different scene representations can be seamlessly integrated into a unified rendering process, greatly reducing integration difficulty and facilitating rapid deployment in actual engines, thereby strongly promoting its widespread application in multiple fields. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a schematic diagram of the structure of a global neural rendering system for a multi-variable hybrid representation scene provided in an embodiment of the present invention;

[0038] Figure 2 This is a flowchart illustrating a global neural rendering method for a multi-variable hybrid representation scene provided in an embodiment of the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0040] The inventive concept of this invention is as follows: Existing fusion-based neural rendering methods often face problems such as model-rendering structure mismatch, scheduling bottlenecks caused by pipeline heterogeneity, and data representation redundancy when working in conjunction with real-time graphics engines, making it difficult to simultaneously achieve image quality, rendering performance, and system scalability. Therefore, to address the problem of the difficulty in efficiently integrating existing neural network rendering technologies with traditional graphics rendering pipelines, embodiments of this invention provide a global neural rendering method and system for multi-faceted hybrid representation scenes. Through modular design, it integrates multi-faceted scene data, utilizes a unified interface and scheduling structure, breaks down the barriers between traditional and neural rendering, achieves efficient collaboration, improves image quality and system flexibility, and reduces integration difficulty.

[0041] like Figure 1 As shown, the embodiment provides a global neural rendering system for multi-variable hybrid representation scenes, including:

[0042] Scene Representation Module: This module is used to collect or load basic geometry, material, and lighting information of a 3D scene, and construct hybrid scene data that includes traditional explicit representations (such as meshes, textures, and material parameters) and neural field representations (such as Gaussian points, volume data, or neural latent space encoding). This module supports configuration management to distinguish which objects or regions use traditional explicit representations and which use neural field representations.

[0043] Geometry Module: In a deferred rendering architecture, the geometry phase typically renders objects in the scene using unlit materials and then writes the geometric information of the objects into the corresponding rendering targets. This module can use rasterization or ray tracing methods to generate a geometry buffer (G-Buffer) containing geometric information and material properties for traditional scene objects. For the neural representation part, methods such as projection sampling, Gaussian sputtering, or volumetric path sampling can be used to generate a G-Buffer containing intermediate rendering features, including but not limited to traditional rendering targets such as normals, tangents, depth, reflectivity, roughness, metallicity, material type, emissivity, and motion vectors. It will also generate neural network intermediate rendering features compatible with traditional data as needed. Finally, a multi-channel G-Buffer output containing geometric information, material properties, and intermediate rendering features is obtained.

[0044] Lighting Module: This module serves as a crucial link between traditional lighting calculations and neural network inference, primarily consisting of one or more neural network sub-modules. Its input is the output from the geometry module, which the network extracts and reconstructs based on spatial structure awareness. This aims to complete lighting information, enhance details, or replace some computational processes in traditional lighting models. Unlike traditional graphics rendering pipelines that rely on rule-based physical models (such as Phong, GGX, Cook-Torrance, etc.) for pixel-by-pixel lighting calculations, the lighting module provides a learnable computational path, enabling more accurate representation of complex materials, global illumination, and screen-space effects.

[0045] In terms of functional organization, the lighting module can be divided into multiple sub-task networks, each focusing on modeling or enhancing specific categories of lighting features, such as subsurface scattering, optical path reconstruction, or occlusion compensation. For example: (1) Lighting reconstruction sub-network: used to reconstruct lighting components that are difficult to simulate in real time, such as global illumination. By inputting the normals, depth, viewpoint direction and intermediate feature representations generated by the neural network in the geometric data, the neural network can efficiently predict effects such as soft shadows and diffuse indirect light, and replace expensive path tracing or photon mapping processes if performance allows; (2) Material perception enhancement sub-network: Based on surface parameters (roughness, metallicity, reflectivity, etc.) and material type (multi-layer material, mixed material, mask material, etc.), enhance the realism and local consistency of the material, which is suitable for complex reflective surfaces (such as frosted metal, painted surfaces, etc.) or situations where material information is missing in low-resolution models; (3) Screen space effect sub-network: Effects such as screen space reflection, depth of field, motion blur, volumetric fog, etc., which are traditionally achieved through post-processing, can also be partially or completely taken over by the neural network, using spatial context and historical frame information to provide more continuous and artifact-free image output; (4) Dynamic resolution adaptation and super-resolution sub-network: Used to improve the final image quality under low resolution or hardware-limited conditions.

[0046] The scheduling and synchronization module is responsible for coordinating multi-stage computation scheduling and cross-stage data caching within the rendering frame, ensuring efficient collaboration among computation modules under heterogeneous representations, resolution differences, and viewpoint changes. In the lighting module, different sub-networks have varying input data requirements, including the required geometry module channel data type, processing resolution, viewport configuration, and scene representation. Therefore, the system abstracts the data requirements of each sub-module into a unified "request" structure, encapsulates necessary resources and context information, and submits it uniformly to the scheduling and synchronization module for centralized management and fusion scheduling.

[0047] During scheduling, the system performs strategic merging based on request content: for example, merging duplicate requests into a single task, overriding low-resolution versions with high-resolution requests, or uniformly processing different types of scene representations when conditions permit. For different scene representations, the system achieves a unified rendering encoding path across multiple representations by additionally outputting type identifier bits during geometry module processing. Simultaneously, to address the differences in output paths between traditional geometry workflows and neural representations, such as in material calculations, the rendering framework introduces a modular shader template structure, supporting dynamic configuration of the output logic at each stage through a macro definition mechanism. Developers only need to write differentiated processing logic for new types of scene representations to achieve expansion without modifying the main rendering workflow.

[0048] Specifically, regarding the shader template structure, when developers build a corresponding G-Buffer output path for a new scene representation, they only need to add the necessary sampling and processing logic to the corresponding shader. They can then switch the currently active output target via compile-time macros or runtime scheduling, thus avoiding repetitive writing or in-depth modifications to the entire pipeline structure. A shader is a piece of program code that runs on the GPU to perform specific rendering tasks. Taking adding a metallic output channel in 3DGaussian Splatting as an example, if this channel is not yet implemented in the original shader, developers only need to embed the corresponding generation logic into the corresponding shader template and control its activation or deactivation via compile-time macros or runtime parameters. With this mechanism, the newly added output channel can be dynamically activated and generated within the existing rendering pass without introducing a separate rendering pass, thereby improving the scalability of the rendering architecture.

[0049] The entire rendering architecture employs a rendering graph structure as its core scheduling and organization method. By explicitly describing the resource dependencies and execution order between various rendering nodes, it ensures efficient collaboration between the traditional graphics rendering process and the neural network inference process. In this structure, the entire rendering process is understood as a pipeline graph. Each rendering stage (Pass) represents a GPU rendering stage with clearly defined inputs, outputs, shader logic, and resource bindings. Each Pass is viewed as a graph node, and its input and output resources (such as textures and buffers) are connected to other nodes through boundary dependencies, forming a directed acyclic graph (DAG) that clearly reflects the data flow path and computational dependencies. To support heterogeneous representations with diverse components and scalable functionality with the integration of new geometry, lighting, or other modules, this rendering graph structure also supports cross-frame scheduling and inference caching mechanisms. Neural rendering often struggles to complete high-quality inference within a single frame. The system can utilize delayed nodes in the rendering graph to propagate results back across frames or share some intermediate neural network feature representations across multiple frames, reducing redundant computations. This graph-based resource planning is particularly critical in mobile platforms and low-power devices, minimizing the lifecycle of resource-used memory and reducing unnecessary power consumption and latency.

[0050] Output Merging Module: This module merges the output from the lighting module with the traditional rendering output to form the final framebuffer output image, enabling global neural rendering. Merging methods can include direct replacement, weighted image compositing, or depth-based occlusion merging; the specific strategy is configured by the user or controlled by the network.

[0051] Furthermore, in a specific implementation of this invention, a global neural rendering system for the aforementioned multi-variable hybrid representation scene has been built and verified based on the Unreal Engine (UE) platform. This system integrates multiple heterogeneous scene representations and achieves flexible, high-quality real-time image output through modular scheduling. Specifically, 3D Gaussian Splatting (3DGS) is used as a typical form of neural scene representation, and the method described in the paper *Neural Global Illumination via Superposed Deformable Feature Field* is selected as the global illumination module to enhance the realism of the rendering. This method models the rendering task as starting from a set of input signals (G-Buffer, scene objects, lighting parameters, etc.), generating local features based on scene object encoding. Centered on each object, it encodes information such as its relative position, material properties, and pose with other objects / light sources to obtain a "local scene representation" that is object-aware. Then, through deformation mapping and neural feature field construction, each object has an independent neural feature field, which deforms according to its local scene representation to adapt to viewpoint and geometric transformations. Since the feature field of each object is defined in its local space, these geometric output channels need to be transformed from world space to local space to express it relative to the object itself. The feature field and local representation support caching, which can accelerate real-time inference. Finally, a unified input decoding network generates the final global illumination image.

[0052] During system implementation, the required geometric stage output channels are first defined on the global illumination network side. Requests are submitted to the scheduling and synchronization module via a unified encapsulation interface. The scheduling and synchronization module is responsible for summarizing and resolving the output channel requirements of the current frame, and automatically constructing an appropriate rendering path based on the object type and representation. For example, for traditional mesh objects, the system reuses the existing BasePass rendering pipeline in Unreal Engine. For 3DGS-based scene objects, the system independently implements a ComputeShader-based rendering path. This path reads the camera parameters of each frame, projects a Gaussian point set in screen space, performs a splatting operation, and calculates the depth, normals, and high-dimensional features contributed to each pixel. This process is constructed in parallel with the traditional pipeline and registers resource outputs and dependency information in a manner compatible with RDG.

[0053] The UE platform's RDG framework supports graph-based rendering resource descriptions. This means that the input and output resources for each pass are explicitly described during registration, and the system automatically constructs a DAG structure, deferring execution until needed for unified scheduling. In this system, the 3DGS rendering pipeline exists alongside the traditional BasePass, and both are registered in the RDG as sub-paths of the geometry module. During execution, the global illumination network reads the corresponding texture resources on demand and registers the output to the merged output module.

[0054] Finally, the merged output module writes the output of the global illumination network into the frame buffer.

[0055] Based on the same inventive concept, such as Figure 2 As shown, this embodiment of the invention also provides a global neural rendering method for multi-variable hybrid representation scenes, including the following steps:

[0056] S1, using the scene expression module to load multi-dimensional mixed scene data;

[0057] S2 utilizes the geometry module to generate a multi-channel geometry buffer based on multi-variable mixed scene data;

[0058] S3, using the lighting module to perform lighting calculations based on a multi-channel geometric buffer through at least one neural network sub-module;

[0059] S4 utilizes the scheduling and synchronization module to coordinate cross-module data flow through a rendering graph structure, and performs resource merging and task scheduling based on a unified request structure.

[0060] S5 utilizes the merged output module to fuse neural network lighting output with traditional rendering output to generate the final frame image to complete global neural rendering.

[0061] Based on the same inventive concept, embodiments of the present invention also provide an electronic device, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-described global neural rendering method for multi-variable hybrid representation scenes when executing the computer program.

[0062] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the above-described global neural rendering method for multi-variable hybrid representation scenes.

[0063] It should be noted that the global neural rendering method, electronic device, and computer-readable storage medium for multivariate mixed representation scenes provided in the above embodiments all belong to the same inventive concept as the global neural rendering system for multivariate mixed representation scenes. For details of their specific implementation process, please refer to the embodiments of the global neural rendering system for multivariate mixed representation scenes, which will not be repeated here.

[0064] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A global neural rendering system for multi-dimensional hybrid representation scenes, characterized in that, include: The scene representation module is used to load multi-variable hybrid scene data, including traditional explicit representations and neural field representations; The geometry module, which is connected to the scene representation module, is used to generate a multi-channel geometry buffer based on multi-source mixed scene data. The lighting module is communicatively connected to the geometry module and contains at least one neural network submodule for performing lighting calculations based on a multi-channel geometry buffer. The scheduling and synchronization module communicates with the geometry and lighting modules, coordinates cross-module data flow through the rendering graph structure, and performs resource merging and task scheduling based on a unified request structure. Specifically, the scheduling and synchronization module performs the following operations: encapsulates the data requirements of sub-modules into a unified request structure, merges duplicate requests and overwrites low-resolution requests with high-resolution requests, and implements unified rendering encoding for multi-dimensional mixed scene representation through type identifier bits. The merged output module is connected to the lighting module and is used to fuse the neural network lighting output with the traditional rendering output to generate the final frame image to complete the global neural rendering.

2. The global neural rendering system for multi-variable hybrid representation scenes according to claim 1, characterized in that, The multi-channel geometry buffer includes: Based on traditional explicit representation using geometric information and material properties generated through methods including rasterization or ray tracing; The neural field representation is based on intermediate rendering features generated using methods including projection sampling, Gaussian sputtering, or volumetric path sampling.

3. The global neural rendering system for multi-variable hybrid representation scenes according to claim 1, characterized in that, The illumination module includes at least one of the following neural network sub-modules: Lighting reconstruction subnetwork, material perception enhancement subnetwork, screen space effects subnetwork, dynamic resolution adaptation and super-resolution subnetwork.

4. The global neural rendering system for multi-variable hybrid representation scenes according to claim 1, characterized in that, The geometry module adopts a modular shader template and dynamically configures the output logic through macro definitions. When adding a new scene representation type, only the differential shader code needs to be extended without refactoring the rendering pipeline.

5. The global neural rendering system for multi-variable hybrid representation scenes according to claim 1, characterized in that, In the rendering graph structure, each rendering stage is regarded as a graph node, and the input and output resources of each graph node are connected to other nodes through boundary dependencies to form a directed acyclic graph.

6. A global neural rendering method for a multi-variable hybrid representation scene, implemented using the global neural rendering system for a multi-variable hybrid representation scene as described in any one of claims 1 to 5, characterized in that, Includes the following steps: S1, using the scene expression module to load multi-dimensional mixed scene data; S2 utilizes the geometry module to generate a multi-channel geometry buffer based on multi-variable mixed scene data; S3, using the lighting module to perform lighting calculations based on a multi-channel geometric buffer through at least one neural network sub-module; S4 utilizes the scheduling and synchronization module to coordinate cross-module data flow through a rendering graph structure, and performs resource merging and task scheduling based on a unified request structure. S5 utilizes the merged output module to fuse neural network lighting output with traditional rendering output to generate the final frame image to complete global neural rendering.

7. An electronic device comprising a memory and one or more processors, the memory being used to store a computer program, characterized in that, The processor is used to implement the global neural rendering method for multi-variable hybrid representation scenes as described in claim 6 when executing a computer program.

8. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by a computer, it implements the global neural rendering method for multi-variable hybrid representation scenes as described in claim 6.

Images