A global neural rendering method and system for hybrid representation optimization

Abstract

This invention discloses a global neural rendering method and system for hybrid representation optimization, belonging to the field of computer graphics technology. The method includes: dividing the screen into multiple tiles; obtaining a first workload index for a first rendering stage on each tile and a second workload index for a second rendering stage on each tile; generating a unified scheduling queue across tiles and stages based on the first and second workload indices; and executing rendering tasks on different tiles in parallel according to the unified scheduling queue; triggering subsequent feature fusion and neural global illumination rendering stages when inter-stage dependency conditions are met to generate an output image with global illumination effects. This invention can significantly improve end-to-end frame rates and reduce tail latency for modern AI accelerators / AI-GPUs while ensuring visual quality and temporal stability. It is suitable for applications such as real-time global illumination, large-scale complex scene rendering, and interactive neural rendering.

Description

Technical Field

[0001] This invention belongs to the field of computer graphics technology, specifically relating to a global neural rendering method and system for hybrid representation optimization. Background Technology

[0002] In recent years, neural rendering has gradually transitioned from implicit representations centered on continuous fields (such as NeRF) to a paradigm that blends explicit and implicit representations with multiple representations. Triangular meshes provide stable first-order visibility and geometric priors, while explicit primitives such as 3D Gaussians compactly depict translucent and microstructural details like hair, fabric, and vegetation. Neural networks handle the approximation and compensation of complex lighting effects, achieving a trade-off between quality and efficiency. Practice shows that the mesh handles the determinism of the main structure and occlusion, while 3D Gaussians use analytical sputtering to complete high-frequency details and translucent phenomena. The combination of these two approaches maintains better multi-view consistency and temporal stability than a single approach, and provides more expressive conditions and feature inputs for subsequent neural global illumination (GI).

[0003] To achieve hybrid representations in real-time rendering, pipelines typically employ a staged mapping strategy: upstream, a geometry buffer (G-buffer) is generated using mesh geometry and rasterization; subsequently, differentiable sputtering is performed based on Gaussian geometry and rasterization, outputting tile-level contributions; finally, the two types of results are fused into a unified intermediate representation, which is then passed to a neural shading / relighting network to predict the final pixel. However, the hardware resource requirements of each stage differ significantly: geometry processing and rasterization are more bandwidth-constrained, while neural shading and global illumination are more vector / tensor computationally constrained. This divergence in resource requirements makes it difficult for static resource mapping strategies to enable each stage to synergistically achieve peak hardware performance.

[0004] The trend towards low-precision computing in modern GPUs is further impacting the system design of neural rendering. New-generation GPUs offer significantly higher peak tensor / matrix unit performance in low-precision formats such as BF16 / FP16 / FP8 than FP32. If the workload can tolerate mixed precision, computational and bandwidth pressure can be significantly reduced without sacrificing quality. However, in practical mixed pipelines, using FP32 precision across multiple stages can lead to long periods of idle tensor units. Besides precision, uneven spatial load is another key factor contributing to reduced tail latency and occupancy: when counted on a tile-by-tile basis, the number of Gaussians per tile exhibits a heavy-tailed distribution, causing an imbalance between fence waiting and scheduling, ultimately hindering frame-time stability.

[0005] In summary, when mapping mesh raster, 3D Gaussian sputtering, and neural global illumination to contemporary GPUs in the hybrid representation pipeline, three common bottlenecks exist: First, overuse of full precision leads to idle tensor units: the continued use of FP32 features in multiple stages causes Tensor Cores to idle for extended periods, preventing throughput from being released; second, spatial load is highly skewed: the number of Gaussians per tile is distributed in a heavy-tailed manner, forming long tails and fences, reducing parallelism; third, the resource demands of each stage are differentiated: geometry / raster is more constrained by bandwidth, while neural shading / global illumination is more constrained by vector / tensor computing power, and static mapping is prone to resource mismatch and idleness.

[0006] To address the aforementioned challenges, current industry solutions rely solely on fixed-function real-time (RT) kernels combined with denoising techniques. While this achieves realistic results, it requires complex sampling-denoising tradeoffs and incurs high hardware costs for dynamic, high-frequency content. Furthermore, simply implementing hybrid precision / kernel fusion and basic scheduling on the software stack often results in intermittent tensor path "lighting up" due to narrow GEMMs, type conversions, splicing / copying overhead, and uneven tile load, making it difficult to stably approach peak values. Therefore, there is an urgent need to optimize the global neural rendering method in the hybrid representation pipeline to systematically address these bottlenecks and achieve high-quality, high-throughput real-time neural rendering. Summary of the Invention

[0007] In view of the above, the purpose of this invention is to provide a global neural rendering method and system for hybrid representation optimization. By constructing a hybrid representation rendering pipeline of mesh and 3D Gaussian, adopting tile-level workload measurement and cross-stage unified scheduling strategy, combining precision-aware hybrid precision execution, and tile-aware adaptive scheduling, early culling, kernel organization and asynchronous optimization mechanism, it significantly improves the end-to-end frame rate and reduces tail latency for modern AI accelerators / AI-GPUs while ensuring visual quality and temporal stability. It is suitable for application scenarios such as real-time global illumination, large-scale complex scene rendering, and interactive neural rendering.

[0008] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0009] In a first aspect, the present invention provides a global neural rendering method for hybrid representation optimization, comprising the following steps:

[0010] The screen is divided into multiple tiles, and the first workload index of the first drawing stage on each tile and the second workload index of the second drawing stage on each tile are obtained.

[0011] Based on the first and second workload metrics, a unified scheduling queue is generated across tiles and stages, and drawing tasks on different tiles are executed in parallel according to the unified scheduling queue.

[0012] When the inter-stage dependency conditions are met, the subsequent feature fusion and neural global illumination rendering stages are triggered to generate an output image with global illumination effects.

[0013] Preferably, the first rendering stage involves soft rasterization of the grid geometry data to generate a geometry buffer, and the first workload metric includes the number of grids or pixels covered by the tiles.

[0014] The second rendering stage involves projecting and sputtering a 3D Gaussian set to generate a Gaussian buffer. The second workload metric includes the number of Gaussians covered by the tiles.

[0015] Preferably, generating a unified scheduling queue across tiles and stages includes:

[0016] Sorting or binning is performed based on the first and second workload indicators of each tile to build a unified scheduling queue that takes into account both the load distribution between tiles and the task association between stages.

[0017] The system prioritizes dispatching heavily loaded tiles with workloads exceeding a preset threshold to a unified scheduling queue, or splits heavily loaded tiles into multiple subtasks for parallel processing.

[0018] Preferably, the inter-stage dependency condition is implemented in the following way:

[0019] The completion status of the first and second rendering stages is asynchronously tracked on a tile-by-tile basis. When it is detected that both stages on the same tile have been completed, the subsequent feature fusion and neural global illumination rendering stages of that tile are triggered, enabling the subsequent stages of different tiles to be executed asynchronously and in parallel.

[0020] Preferably, it also includes employing a precision-aware scheduling strategy:

[0021] Different computational precisions are assigned to different rendering stages or different data types. Specifically, the first precision is used for geometry-related calculations, the first precision or a second precision lower than the first precision is used for Gaussian mixture, feature fusion and neural global illumination rendering, and the second precision or a third precision lower than the second precision is used for intermediate activations or final output.

[0022] Preferably, it also includes employing an early rejection strategy:

[0023] Before dispatching the second drawing phase drawing task, the visibility information generated in the first drawing phase is used to skip the second drawing phase drawing contributions that are occluded or whose contributions are below the threshold within the corresponding tiles.

[0024] Preferably, it also includes kernel organization and asynchronous optimization strategies:

[0025] Kernel fusion is performed on computations across operators or stages, and data layout alignment is optimized.

[0026] Data within the tile is prefetched to the on-chip high-speed memory via asynchronous instructions, and the first rendering stage, the second rendering stage, and the subsequent feature fusion and neural global illumination rendering stages are executed in overlapping streams, with dependencies guaranteed by events / fences.

[0027] Secondly, embodiments of the present invention also provide a global neural rendering system optimized for hybrid representation, implemented using the aforementioned global neural rendering method optimized for hybrid representation, comprising: a tile load acquisition module, a unified scheduling execution module, and a fusion inference output module;

[0028] The tile load acquisition module is used to divide the screen into multiple tiles, and to obtain the first workload index on each tile in the first drawing stage and the second workload index on each tile in the second drawing stage.

[0029] The unified scheduling execution module is used to generate a unified scheduling queue across tiles and stages based on the first workload index and the second workload index, and to execute drawing tasks on different tiles in parallel according to the unified scheduling queue.

[0030] The fusion inference output module is used to trigger the subsequent feature fusion and neural global illumination rendering stages when the inter-stage dependency conditions are met, so as to generate an output image with global illumination effects.

[0031] Thirdly, embodiments of the present invention also provide an electronic device, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-described global neural rendering method for hybrid representation optimization when the computer program is executed.

[0032] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a computer, implements the aforementioned global neural rendering method optimized for hybrid representation.

[0033] Compared with the prior art, the beneficial effects of the present invention include at least the following:

[0034] (1) This invention divides the work volume of each stage on each tile by tile partitioning and obtaining the workload indicators of each stage, and generates a unified scheduling queue across tiles and stages. This enables load awareness and unified scheduling of heterogeneous tasks in the grid and Gaussian regions, which can effectively solve the spatial load skew problem and significantly reduce the impact of long-tail tiles on frame time. By combining staged mixed precision, tile-aware scheduling with kernel organization and asynchronous optimization, this invention can reduce end-to-end frame time by about 20%-27% in complex scenarios compared to the full FP32 baseline.

[0035] (2) By assigning different precision to different stages or data types, the present invention retains FP32 for geometric visibility key quantities and uses low precision such as BF16 / INT8 for neural GI and feature processing, which can significantly reduce memory bandwidth and intermediate buffer usage while maintaining high-quality global illumination and timing stability, and improve the utilization of streaming multiprocessors and tensor units.

[0036] (3) This invention further reduces redundant computation by tile-level early culling, and reduces data transfer overhead and kernel startup latency between stages by kernel fusion across operators / stages, data layout optimization and asynchronous prefetching and streaming overlap, so that the hybrid rendering pipeline can maintain high efficiency and stability in large-scale scenarios, and finally achieve a balance between accuracy and efficiency. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart illustrating the global neural rendering method for hybrid representation optimization provided in an embodiment of the present invention.

[0039] Figure 2 This is a schematic diagram of the overall framework of the global neural rendering method for hybrid representation optimization provided in the embodiments of the present invention;

[0040] Figure 3 This is a schematic diagram of the tile workload distribution and long-tail load problem provided in an embodiment of the present invention;

[0041] Figure 4 This is a schematic diagram of the software architecture and runtime orchestration of the global neural rendering method for hybrid representation optimization provided in an embodiment of the present invention;

[0042] Figure 5 These are schematic diagrams illustrating three representative scenarios / use cases provided in embodiments of the present invention;

[0043] Figure 6 This is a schematic diagram of the structure of a global neural rendering system for hybrid representation optimization provided in an embodiment of the present invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0045] The inventive concept of this invention is as follows: Addressing the system-level problems of high end-to-end latency, severe spatial load skew, insufficient utilization of low-precision units, and large data transport overhead between stages in existing hybrid representation neural rendering scenarios under real-time global illumination, this invention provides a global neural rendering method and system optimized for hybrid representation. By dividing the screen into tiles and obtaining workload indicators for different rendering stages on each tile, a unified scheduling queue is generated across tiles and stages based on the workload indicators, achieving load awareness and unified scheduling for heterogeneous tasks. Simultaneously, a staged hybrid precision strategy is adopted to allocate different precisions to different rendering and inference stages, combined with early culling, kernel fusion, and asynchronous prefetch optimization, significantly reducing end-to-end latency, alleviating load skew, improving computing resource utilization, and reducing data transport overhead.

[0046] like Figure 1 and Figure 2 As shown, the embodiment provides a global neural rendering method for hybrid representation optimization, including the following steps:

[0047] S1, divide the screen into multiple tiles, and obtain the first workload index on each tile in the first drawing stage, and the second workload index on each tile in the second drawing stage.

[0048] In this embodiment, initialization and resource preparation are performed first: inputting camera pose parameters and determining viewpoint information; loading mesh geometry data (vertices / indexes / materials), 3D Gaussian sets (position, covariance / scale, color / features, transparency, etc.), lighting parameters, and trained neural network weights; setting tile partitioning parameters according to the target resolution to divide the screen into tiles of fixed or adaptive sizes (such as 16×16 or 32×16); pre-allocating and reusing the following buffers in global video memory: geometry buffer, tile statistics buffer, tile → Gaussian meta-list buffer, Gaussian buffer, fusion feature buffer, network intermediate activation buffer, and frame output buffer; establishing a multi-stream execution environment and creating event / fence objects for cross-stage dependencies.

[0049] The first rendering stage involves soft rasterizing the mesh geometry data to generate a geometry buffer. Specifically, on the GPU, mesh vertex transformation, clipping, and triangle traversal are performed. A soft raster approach is used to calculate the triangle coverage probability and depth visibility at each pixel / sampling point, generating a geometry buffer. This buffer contains at least depth, normals, and material / texture parameters, and optionally includes view / lighting-related intermediates. In terms of data flow, the mesh geometry data buffer is stored in global video memory (DRAM). After being divided into tiles, it is read into on-chip shared memory (SPM / Shared Memory) in blocks for edge function / coverage probability, depth, and attribute interpolation calculations. This yields the geometry buffer block for that tile, which is then written back to global video memory for consumption in subsequent feature fusion and neural global illumination rendering stages. In the control flow, a scheduling module generates a raster task descriptor for each tile, specifying the input buffer address, tile range, and output geometry buffer write-back location. Events / fences trigger the second rendering stage or feature fusion stage upon completion of the first rendering stage. Simultaneously, the first workload indicators for the first rendering stage include the number of grids covered by the tiles (e.g., the number of triangles) or the number of pixels. This is obtained by statistically analyzing the grid coverage of each tile during each frame's runtime and writing it into the tile statistics buffer. To ensure numerical stability, the first rendering stage uses FP32 precision (first precision) for key geometric visibility quantities (e.g., vertex coordinates, edge function coefficients, depth / occlusion determination).

[0050] The second rendering stage involves projecting and sputtering a 3D Gaussian set to generate a Gaussian buffer. Specifically, view-related projection, screen-space bounding box calculation, and tile assignment are performed on the 3D Gaussian primitives. The Gaussian workload for each tile is calculated, and a tile → Gaussian primitive list is generated. Subsequently, the tiles are sorted by depth or keyword and Gaussian sputtering / blending is performed to obtain the Gaussian buffer, which at least includes the Gaussian contribution of color / feature accumulation and transparency, and optionally includes Gaussian normals / depth, etc. In terms of data flow, the 3D Gaussian set is stored in global video memory. The Gaussian parameters of each tile can be prefetched in blocks to on-chip shared memory and vectorized for blending. The blending result is written back to the corresponding Gaussian buffer block of the tile and then back to global video memory. In the control flow, the second workload index of the second rendering stage is obtained through the scheduling module, including the number of Gaussians or the number of pixels covered by the tile. This is obtained by counting the number of Gaussian primitives of each tile every frame and writing it to the tile statistics buffer. To reduce computational overhead and memory bandwidth usage while ensuring computational accuracy, the second rendering stage uses BF16 precision (second precision) for Gaussian projection, sputtering, and blending.

[0051] S2 generates a unified scheduling queue across tiles and stages based on the first and second workload metrics, and executes drawing tasks on different tiles in parallel according to the unified scheduling queue.

[0052] like Figure 3 The figure illustrates the workload distribution across screen space tiles (e.g., the number of Gaussians involved per tile). The horizontal axis represents the tile indices sorted by workload, and the vertical axis represents the number of Gaussians or the number of mesh triangles. As can be seen, a small number of tiles contain extremely high workloads (e.g., tens of thousands of Gaussians), while most tiles have lower workloads, exhibiting a clear heavy-tailed distribution. This distribution causes long-tailed tiles to become the bottleneck for overall frame time—even if most tiles complete quickly, a few heavily loaded tiles can still slow down the entire frame rendering. To address this issue, in this embodiment, the scheduling module sorts or buckets tiles based on a first and second workload metric, generating a tile task queue (tile_map) and a task descriptor for each tile. This constructs a unified scheduling queue that considers both the load distribution between tiles and the task association between stages. Based on this unified scheduling queue, tiles with a workload greater than or equal to a preset threshold (e.g., 10) are prioritized for assignment. 4 For heavily loaded tiles, the process can either split them into multiple subtasks for parallel processing, or allow them to be processed in batches or merged if the workload is less than the preset threshold. This reduces scheduling overhead and improves throughput. The first and second drawing phases can proceed in parallel in different streams / queues, and data consistency for the same tile can be ensured through lightweight synchronization.

[0053] Furthermore, before dispatching the second rendering stage, the visibility information generated in the first rendering stage (such as depth / occlusion masks in the geometry buffer) is used to skip the second rendering stage's rendering contributions within the corresponding tiles that are occluded or whose contributions are below a threshold (specifically, the depth value of the first 3D Gaussian index at that location). This achieves tile-level early culling, reducing redundant mixed computation and bandwidth overhead. This early culling is implemented at the tile level, filtering Gaussian occlusions before Gaussian sputtering, which can significantly reduce redundant computation and bandwidth in dense scenes.

[0054] S3 triggers subsequent feature fusion and neural global illumination rendering stages when the inter-stage dependency conditions are met, in order to generate an output image with global illumination effects.

[0055] In this embodiment, the inter-stage dependency condition is implemented in the following way: the completion status of the first and second drawing stages is tracked asynchronously on a tile-by-tile basis. Tile-level events or atomic counters can be used to record the stage completion status of each tile. When the scheduling module detects that both stages on the same tile have been completed, the subsequent feature fusion and neural global illumination drawing stages of that tile are immediately triggered, so that the subsequent stages of different tiles can be executed asynchronously and in parallel, avoiding pipeline blockage caused by global synchronization.

[0056] The feature fusion stage aligns and merges the geometric buffers generated in the first rendering stage and the Gaussian buffers generated in the second rendering stage at the tile granularity into a unified intermediate feature representation: aligning grid attributes and Gaussian contributions by pixel position, performing feature concatenation, weighting, and normalization to generate the compact input tensor required for neural global illumination. In terms of data flow, the feature fusion stage reads the geometric buffers and Gaussian buffers from global memory (DRAM), concatenates and reduces pixel blocks within the same tile on-chip, and writes the fused features back to the fused feature buffer in DRAM. In terms of control flow, the feature fusion stage is triggered by the scheduling module when the first and second rendering stages for the corresponding tile are completed, avoiding global synchronization, and using tile-level events or atomic counting for dependency tracking. To reduce bandwidth and type conversion overhead, the feature fusion process can be executed using BF16, and type conversion and concatenation operators can be fused using a kernel.

[0057] The Neural Global Illumination (GNIR) rendering stage uses a lightweight neural network (including a feature extraction network, a relighting / GI network, and an output network) to infer the fused features, predict the color of each pixel with GNIR effects, and write the output color to the frame output buffer. A super-resolution stage can be optionally enabled; when a higher output resolution is required, the output of the GNIR rendering stage is upsampled and enhanced for detail before being written back to the final frame output buffer. In the data stream, the GNIR rendering stage reads the fused feature buffer and generates intermediate activations / output colors. Activations can reuse pre-allocated buffers to reduce memory allocation and copying. The final output is a fully shaded RGB image at the target resolution, with optional output of auxiliary results such as depth / normals. In the control flow, the GNIR rendering stage can be piped parallel to the next frame processing of the first or second rendering stage, and cross-frame dependencies are managed through double buffering / circular buffering. The GNIR rendering stage primarily uses BF16 execution and can store some intermediate activations and the final output of the GNIR network in INT8 (third precision) to reduce memory usage.

[0058] In this embodiment, a precision-aware hybrid precision strategy is employed throughout the aforementioned steps. In the offline phase, low precision (BF16 / INT8) needs to be applied to different stages / attributes (e.g., depth, normals, material properties, Gaussian features, fusion features, network weights, and activations) and their impact on the final image quality needs to be evaluated. This yields a stage-level / attribute-level sensitivity ranking, which, combined with a cost model (memory bandwidth, computing power, startup overhead), generates a precision configuration table. During runtime, the scheduling module selects the corresponding kernel version (FP32, BF16, or INT8) for each stage based on the precision configuration table.

[0059] Furthermore, execution efficiency is further improved through kernel organization and asynchronous optimization. Kernel organization optimization: By optimizing cross-operator / cross-stage kernel fusion (such as fusion type conversion, splicing and normalization) and data layout and alignment (such as explicitly carrying input / output data format and alignment information in the task descriptor, and using 2-channel / 4-channel packaging), bandwidth utilization and tensor / vector instruction throughput are improved, kernel startup and intermediate write-back are reduced, and additional kernel calls caused by implicit type conversion are avoided, thus reducing end-to-end overhead. Asynchronous optimization: Data within tiles (such as Gaussian parameter blocks and fused feature blocks) is prefetched to on-chip high-speed memory through asynchronous instructions, and the first drawing stage, the second drawing stage, and subsequent stages are executed in overlapping streams to hide memory access and startup overhead. Dependencies are established at the tile granularity through events / fences to ensure data consistency.

[0060] like Figure 4 The diagram shown is a schematic diagram of the software architecture and runtime orchestration of the global neural rendering method for hybrid representation optimization provided in an embodiment of the present invention.

[0061] (i) Implementation setup and initialization: This is used to complete the preparation of the runtime environment and the configuration of the output path. Its output is a runtime context that can be reused in subsequent stages.

[0062] 1) Setting up the code environment: This refers to initializing the graphics processor's runtime environment, including selecting the target device, creating a code context, creating a default or multiplexed code stream, initializing the memory allocator / memory pool, and setting the computation mode related to mixed precision (e.g., allowing BF16 / FP16 / FP8 kernel paths). Inputs are the device number and runtime parameters; outputs are code contexts, streams, and event objects that can be reused at each stage.

[0063] 2) Setting Path: This refers to setting the input path for scene resources (mesh file, Gaussian file, camera trajectory, network weights, configuration files, etc.), as well as the log and result output path; its output is unified resource location information, which can be called by sub-modules such as "Load Scene", "Load Model", and "Image Input / Output".

[0064] 3) Create output directory: This refers to creating the output directory structure for this run in the file system (such as frame sequence directory, statistics log directory, and debugging visualization directory) to save the final image, stage time, and intermediate debugging information.

[0065] (ii) Rendering module: used to establish the execution carrier of the end-to-end rendering pipeline and provide the ability to load scenes and obtain running parameters.

[0066] 1) Establishing the rendering process: This refers to instantiating the hybrid representation global neural rendering pipeline object of this invention, creating tile partitioning parameters, initializing the data structure of the cross-stage unified scheduling queue, and requesting and binding video memory resources such as geometry buffer, Gaussian buffer, fusion feature buffer, network activation buffer, and output frame buffer; its output is a rendering pipeline handle that can be called by subsequent "mesh rasterization stage / Gaussian rasterization stage / feature fusion stage / neural network stage".

[0067] 2) Loading Scene: This refers to reading mesh geometry data (vertices, indexes, materials / textures) and 3D Gaussian sets (position, scale / covariance, color or features, transparency, etc.) from the input path, and performing necessary format conversion and compression / alignment (e.g., organizing by tile or block, memory layout), and writing it to global video memory; its output is a mesh data buffer and a Gaussian data buffer, as well as basic metadata (such as bounding boxes, hierarchical indexes) used for tile attribution determination.

[0068] 3) Get variables: This refers to reading or generating runtime variables and control parameters, including resolution, tile size (such as 16×16 or 32×16), stage precision configuration (such as mesh key quantity, Gaussian / fusion), heavy load threshold, early culling threshold, and whether to enable kernel fusion and asynchronous prefetching, etc. Its output is used to drive "unified scheduling execution" and kernel selection at each stage.

[0069] (iii) Model module: used to complete the configuration loading, camera and transformation calculations required for network and rendering, as well as the generation of front-end rendering results for two types of representations (mesh / Gaussian).

[0070] 1) Loading configuration file: This refers to loading the pipeline's configuration file (such as network structure parameters, feature dimensions, quantization / precision strategies, data layout descriptions, debug switches, etc.) and parsing it into a runtime-accessible structure; this configuration file can serve as one of the sources for the "Configuration" submodule.

[0071] 2) Transformation calculation: refers to the computational geometry and Gaussian transformation from world coordinates to screen coordinates, including view matrix / projection matrix, affine transformation matrix and its inverse matrix, etc.; its input is "camera pose" and intrinsic parameters, and the output is the matrix and derived parameters (such as view frustum clipping parameters, screen space scaling factor) used in subsequent "mesh rendering" and "Gaussian rendering".

[0072] 3) Initialization module: refers to the initialization of the shared state between the model side and the rendering side, such as initializing the operator graph of the fusion network / global illumination network, initializing the parameters used for object encoding (such as hyperplane parameters, position encoder parameters), initializing the support vectors and weights of the variable support vector machine module, etc.; its output is the initial state of each sub-network / sub-module.

[0073] 4) Obtain camera pose: This refers to obtaining or updating camera extrinsic parameters (position and orientation) and camera intrinsic parameters (focal length, principal point, distortion, optional), which can come from preset trajectories, sensor inputs, or interactive control; its output is used by "conversion calculation", "camera setup", and "mesh rendering / Gaussian rendering".

[0074] 5) Loading the model: This refers to loading the trained neural network weights (including parameters of the network related to object encoding / representation encoding, feature fusion, and neural global illumination rendering), and converting the weights into storage formats such as BF16 / INT8 as needed; its output is the network weight buffer and operator selection information.

[0075] 6) Mesh Rendering: This refers to the first rendering stage (mesh geometry stage / mesh rasterization stage), where the mesh triangles are soft rasterized on the GPU, pixel coverage, depth, and attribute interpolation are calculated to form a geometry buffer. Its inputs are the mesh data buffer, camera pose, and transformation matrix; its outputs are G-buffer components (including at least depth, normals, and material / texture parameters, and optionally visibility mask / occlusion information), which are written to the geometry buffer in global video memory.

[0076] 7) Geometry buffer generation: refers to the process of organizing and writing the output of "mesh rendering" to disk / buffer, that is, writing the geometric attributes of each tile or pixel to the geometry buffer and generating visibility information that can be used in subsequent stages; its output is directly consumed by "early culling" and "feature fusion stage".

[0077] 8) Gaussian Rendering: This refers to the second rendering stage (Gaussian branch / Gaussian rasterization stage in the Gaussian and mesh geometry stages) performing projection, tile assignment determination, and sputtering blending on a 3D Gaussian set. Its inputs are the Gaussian data buffer, camera pose, and transformation matrix; its outputs are Gaussian buffer components (such as color / feature accumulation, transparency, and optional Gaussian depth / normals), written to the Gaussian buffer in global video memory. Gaussian rendering typically uses BF16 for blending calculations to reduce bandwidth and computational overhead.

[0078] Additional notes: Figure 4 "Establishing a camera" refers to constructing a view matrix and a projection matrix based on the "camera pose" and camera intrinsic parameters, and writing them into the renderer / kernel parameters so that the mesh and Gaussian can complete consistent projection and visibility determination under the same camera. Figure 4 In this context, "sputtering" refers to the process of accumulating the contribution (color or feature) of a Gaussian projected onto the screen space to pixels / sample points according to its coverage area and then blending it according to transparency. It is usually performed in tiles to coordinate with unified scheduling.

[0079] (iv) Data processing module: used to load and preprocess data in training / evaluation or offline verification scenarios; it can also be used to load reference images or camera trajectories in pure online rendering scenarios.

[0080] 1) Loading dataset: refers to loading a set of scene data or test sequences (including multi-view images, camera trajectories, lighting parameters, etc.) for training / validation, and its output is an iterable data index.

[0081] 2) Loading data: This refers to loading the required data items (such as the current frame's camera pose, reference image, and lighting conditions) frame by frame or batch by batch, and copying the necessary data to the video memory.

[0082] 3) Preprocessing data: This refers to normalizing, aligning coordinate systems, scaling, preparing tiles, and converting formats of the input data to meet the interface requirements of subsequent "forward inference" and "image input / output".

[0083] (v) Reasoning and output module: used to perform feature fusion and neural global illumination reasoning, and to complete result output and performance observation.

[0084] 1) Forward inference: This refers to the forward computation process in the neural network stage, including input processing of the fused features, inference of the color / lighting result for each pixel through the neural global illumination rendering network, and optional output synthesis (e.g., overlay with direct light / primary color, tone mapping, etc.). Its input is the fused feature buffer and network weights; its output is the final color frame buffer.

[0085] 2) Time observation: refers to using a "timer" to perform time statistics on sub-stages (such as mesh rasterization, Gaussian sputtering, feature fusion, neural network inference, etc.), and output frame time, tail delay and tile-level time distribution to verify the effect of unified scheduling and optimization strategy.

[0086] 3) Image input and output: This refers to writing the final output image to the output directory, and optionally outputting intermediate results (geometric buffer, Gaussian buffer, fusion feature visualization) and reading reference input (such as a reference image for comparison).

[0087] (vi) Hybrid Rendering Neural Network Generator: For ease of engineering implementation, Figure 4 The network-related construction and auxiliary functions are abstracted into a "hybrid rendering neural network generator", which is used to generate / manage the network components required for feature fusion and neural global illumination rendering, and shares camera and primitive information with the renderer.

[0088] 1) Initialization & Configuration & Assistance: Includes sub-items such as "Configuration, Renderer, Gaussian Primitives, Timers, Camera Pose, and Debugging".

[0089] Among them, "Configuration" describes the parameters related to network structure, feature dimensions, accuracy configuration, tile partitioning, and unified scheduling; "Renderer" refers to the rendering pipeline object that performs mesh rasterization and Gaussian sputtering; "Gaussian primitives" refer to a single Gaussian element in a 3D Gaussian set (including position, scale / covariance, color / feature, transparency, etc.); "Timer" is used to count the time taken at each stage; "Camera pose" is used to drive projection and transformation; and "Debug" is used to output logs, check intermediate features, and visualize results.

[0090] 2) Key Components: These include "object encoding (hyperplane, position encoder), variable support vector machine (super support vector machine, weights), object representation encoding (support vector machine, transform fusion), and decoder (support vector machine, color)". These components can be implemented as follows:

[0091] (a) Object Encoding: Used to encode the spatial location, geometric attributes, or semantic identifiers of an object into a compact vector. The position encoder is used to perform frequency domain / multi-scale encoding on continuous coordinates; the hyperplane is used to parametrically describe the partitioning / discrimination boundaries of the space or feature space, thereby providing conditions for subsequent feature aggregation. The output of the object encoding is used as an object-level conditional vector to participate in feature fusion and neural illumination inference.

[0092] (b) Variable Support Vector Machine: To achieve lightweight feature mapping and adaptive representation, a kernel function approximation or basis function expansion module consisting of several "support vectors" and "weights" can be used. A super support vector machine is used to represent an extended form of the support vector set (e.g., parameterization of multiple sets / multi-head support vectors or higher-dimensional support vectors) to adaptively select or reweight on different tiles / objects; the weights are used to perform a weighted summation of the support vector responses, and the output serves as an intermediate feature.

[0093] (c) Object Representation Encoding: This encoding is used to fuse the geometric / appearance information generated by the mesh branch and the Gaussian branch into an object-level representation. This encoding can reuse the support vector machine module for feature projection and fuse the geometric transformation features obtained from affine / shape transformations with the rendering features through "transform fusion" to output a unified object representation vector.

[0094] (d) Decoder: Used to decode the fused object / tile-level features into renderable output (such as color or lighting residuals). "Color" refers to the pixel of the decoded output or its increment; the decoder can be implemented using a lightweight mapping in the style of support vector machines and a combination of linear / non-linear methods.

[0095] 3) Transformations: These include "Gaussian shape transformation (scaling, rotation, translation), affine transformation (matrix multiplication), and shape transformation (scaling / rotation, translation)". Specifically, Gaussian shape transformation is used to transform the scale / covariance and center of a Gaussian object from the world system to the camera / screen space; affine transformation achieves coordinate transformation through matrix multiplication; and shape transformation is used to scale, rotate, and translate the local coordinates of mesh vertices or objects to match the current camera pose and scene animation.

[0096] (seven) Figure 4 The meaning of the right-hand phase box and its interface relationship: Figure 4 The stage boxes visually illustrate the positions of mesh, Gaussian, feature fusion, and neural global illumination rendering in the end-to-end pipeline.

[0097] 1) Gaussian and Mesh Geometry Stage: This stage involves reading, transforming, clipping the view frustum, and determining tile assignment for the geometric data of the mesh and Gaussian grids. Its output includes tile-level workload statistics and projection parameters that can be used for subsequent rasterization.

[0098] 2) Mesh rasterization stage: Corresponding to the first rendering stage, soft rasterization is performed to generate the geometry buffer. The inputs are mesh data and camera parameters; the outputs are the geometry buffer and visibility information, which are written to the geometry buffer.

[0099] 3) Gaussian rasterization stage: Corresponding to the second rendering stage, Gaussian rasterization is performed by projection and sputtering to generate a Gaussian buffer. The input is Gaussian primitives and camera parameters; the output is the Gaussian buffer, which is then written to the Gaussian buffer.

[0100] 4) Feature Fusion Stage: This stage involves aligning the geometric buffer and Gaussian buffer at the tile / pixel granularity, then concatenating, weighting, normalizing, and compressing them to obtain a fused feature buffer. The inputs are the geometric buffer and Gaussian buffer; the output is a fused feature tensor, which is written into the fused feature buffer.

[0101] 5) Neural Network Stage: This stage refers to performing forward inference on the fused features. "Input processing" includes formatting / normalizing / quantizing and tensor packing of the fused features, while "output synthesis" includes compositing the network output with the rendering substrate (e.g., overlaying direct light, performing tone mapping, etc.) to output the final image.

[0102] 6) Rendering & Inference & Shading: Figure 4 In this context, "Rendering & Inference" refers to the unified scheduling and execution module orchestrating each stage across tiles and stages; "Shading" refers to the completion of feature fusion and global illumination calculation in the neural network stage, where "Calculate Global Illumination" refers to predicting global illumination effects such as indirect lighting, soft shadows, and mutual reflection.

[0103] 7) Inter-stage dependencies and triggering relationships: For the same tile, once both the mesh rasterization stage and the Gaussian rasterization stage are completed (which can be recorded through tile-level events / counters / fences), the feature fusion and neural network inference of that tile are immediately triggered, enabling subsequent stages of different tiles to be executed asynchronously and in parallel; at the same time, the visibility information of the geometry buffer can be used to perform early culling of Gaussian sputtering to reduce redundant calculations.

[0104] like Figure 5 As shown, to further illustrate the effectiveness of the method of the present invention, three typical application scenarios are provided for illustrative purposes.

[0105] In indoor mixed detail scenes, regular solid structures such as desktops and furniture are represented by meshes and stable geometric buffers are generated through the first rendering stage. High-frequency details such as dolls and hair are represented by Gaussians and semi-transparent or detail contributions are generated by sputtering through the second rendering stage. The subsequent feature fusion and neural global illumination rendering stages predict soft shadows and indirect lighting based on the fused features, so that hair and fabric produce visible projections and diffuse reflections on the surrounding surfaces, demonstrating the synergistic advantages of mixed representation and neural global illumination.

[0106] In scenes with complex materials and contact shadows, materials such as leather, carpets, and cushions exhibit obvious contact shadows and indirect reflections. The large-scale shapes and stable visibility are depicted by the mesh part in the first rendering stage, while the fabric details and microstructures are depicted by the Gaussian part in the second rendering stage. Furthermore, by using tile-aware scheduling, tiles are subdivided or prioritized in Gaussian-dense areas to avoid slowing down the entire frame in long-tailed areas, thus ensuring the real-time rendering of complex materials.

[0107] In large-scale Gaussian scenes outdoors, the large area of ​​grass and a large number of small objects result in a huge number of Gaussian tiles with uneven distribution height, exhibiting obvious heavy-tail workload characteristics. This invention can significantly reduce redundant mixed computation and tail latency by prioritizing the dispatch of heavy-load tiles and combining early culling, keeping the end-to-end frame rate within an interactive range, thus verifying the efficiency and stability of this invention in large-scale sparse scenes.

[0108] In summary, the global neural rendering method for hybrid representation optimization provided by this invention constructs an end-to-end pipeline from the mesh geometry processing and soft raster stage, the Gaussian projection / sorting and sputtering stage, the tile-level feature fusion stage, the neural global illumination inference stage, and the optional super-resolution and output stage. Each stage is implemented on an AI accelerator with a general-purpose computing kernel, avoiding reliance on offline pre-computed geometry buffers or fixed-function shading channels. Ultimately, it can output a fully shaded RGB image at the target resolution (with optional output of auxiliary results such as depth / normals), significantly improving the end-to-end frame rate while ensuring high-quality global illumination effects.

[0109] Based on the same inventive concept, such as Figure 6 As shown, this embodiment of the invention also provides a global neural rendering system 600 for hybrid representation optimization, including: a tile load acquisition module 610, a unified scheduling execution module 620, and a fusion inference output module 630.

[0110] The tile load acquisition module 610 is used to divide the screen into multiple tiles and obtain the first workload index on each tile in the first drawing stage and the second workload index on each tile in the second drawing stage.

[0111] The unified scheduling execution module 620 is used to generate a unified scheduling queue across tiles and stages based on the first workload index and the second workload index, and to execute drawing tasks on different tiles in parallel according to the unified scheduling queue.

[0112] The fusion inference output module 630 is used to trigger subsequent feature fusion and neural global illumination rendering stages when the inter-stage dependency conditions are met, so as to generate an output image with global illumination effects.

[0113] Based on the same inventive concept, embodiments of the present invention also provide an electronic device, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-described global neural rendering method for hybrid representation optimization when executing the computer program.

[0114] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the above-described global neural rendering method optimized for hybrid representation.

[0115] It should be noted that the global neural rendering system, electronic device, and computer-readable storage medium for hybrid representation optimization provided in the above embodiments all belong to the same inventive concept as the global neural rendering method for hybrid representation optimization. The specific implementation process is detailed in the embodiments of the global neural rendering method for hybrid representation optimization, and will not be repeated here.

[0116] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A global neural rendering method for hybrid representation optimization, characterized in that, Includes the following steps: The screen is divided into multiple tiles, and a first workload index for the first rendering stage on each tile and a second workload index for the second rendering stage on each tile are obtained. The first rendering stage is to perform soft rasterization on the mesh geometry data to generate a geometry buffer. The first workload index includes the number of meshes or pixels covered by the tile. The second rendering stage is to project and sputter a 3D Gaussian set to generate a Gaussian buffer. The second workload index includes the number of Gaussians covered by the tile. Based on the first workload index and the second workload index, a unified scheduling queue is generated across tiles and stages. This includes: sorting or binning each tile based on the first workload index and the second workload index to construct a unified scheduling queue that takes into account the load distribution between tiles and the task association between stages; prioritizing the dispatch of heavily loaded tiles with a workload higher than a preset threshold, or splitting heavily loaded tiles into multiple sub-tasks for parallel processing according to the unified scheduling queue; and executing drawing tasks on different tiles in parallel according to the unified scheduling queue. When the inter-stage dependency condition is met, the subsequent feature fusion and neural global illumination rendering stages are triggered to generate an output image with global illumination effects. The inter-stage dependency condition is implemented in the following way: the completion status of the first rendering stage and the second rendering stage are asynchronously tracked on a tile-by-tile basis. When it is detected that both stages on the same tile have been completed, the subsequent feature fusion and neural global illumination rendering stages of that tile are triggered, so that the subsequent stages of different tiles can be executed asynchronously and in parallel.

2. The global neural rendering method for hybrid representation optimization according to claim 1, characterized in that, This also includes employing precision-aware scheduling strategies: Different computational precisions are assigned to different rendering stages or different data types. Specifically, the first precision is used for geometry-related calculations, the first precision or a second precision lower than the first precision is used for Gaussian mixture, feature fusion and neural global illumination rendering, and the second precision or a third precision lower than the second precision is used for intermediate activations or final output.

3. The global neural rendering method for hybrid representation optimization according to claim 1, characterized in that, This also includes employing an early rejection strategy: Before dispatching the second drawing phase drawing task, the visibility information generated in the first drawing phase is used to skip the second drawing phase drawing contributions that are occluded or whose contributions are below the threshold within the corresponding tiles.

4. The global neural rendering method for hybrid representation optimization according to claim 1, characterized in that, This also includes employing kernel organization and asynchronous optimization strategies: Kernel fusion is performed on computations across operators or stages, and data layout alignment is optimized. Data within the tile is prefetched to the on-chip high-speed memory via asynchronous instructions, and the first rendering stage, the second rendering stage, and the subsequent feature fusion and neural global illumination rendering stages are executed in overlapping streams, with dependencies guaranteed by events / fences.

5. A global neural rendering system optimized for hybrid representations, implemented using the global neural rendering method optimized for hybrid representations as described in any one of claims 1 to 4, characterized in that, include: The module consists of a tile load acquisition module, a unified scheduling and execution module, and a fusion inference output module. The tile load acquisition module is used to divide the screen into multiple tiles, and to obtain the first workload index on each tile in the first drawing stage and the second workload index on each tile in the second drawing stage. The unified scheduling execution module is used to generate a unified scheduling queue across tiles and stages based on the first workload index and the second workload index, and to execute drawing tasks on different tiles in parallel according to the unified scheduling queue. The fusion inference output module is used to trigger the subsequent feature fusion and neural global illumination rendering stages when the inter-stage dependency conditions are met, so as to generate an output image with global illumination effects.

6. An electronic device comprising a memory and one or more processors, the memory being used to store a computer program, characterized in that, The processor is used to implement the global neural rendering method for hybrid representation optimization as described in any one of claims 1 to 4 when executing a computer program.

7. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by a computer, it implements the global neural rendering method for hybrid representation optimization as described in any one of claims 1 to 4.

Images