A global neural rendering method and system based on a programmable rasterization engine
By introducing a programmable rasterization engine into the AI accelerator, the problem of the AI accelerator's inability to uniformly support multi-representation neural rendering tasks is solved, achieving efficient rasterization stage processing and improving rendering performance and frame rate.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2026-04-03
- Publication Date
- 2026-07-31
AI Technical Summary
Existing technologies struggle to efficiently and uniformly support neural rendering tasks with multiple representations, such as meshes and 3D Gaussian splashing, on AI accelerators. The rasterization stage suffers from weak programmability, low data reuse, and high memory access and scheduling overhead.
By introducing a programmable rasterization engine, and combining rasterization instruction parsing and vector micro-operations within the vector core of the AI accelerator with direct memory access units and on-chip shared memory, unified rasterization computation of multiple primitive representations is achieved, reducing the memory access and scheduling overhead of key steps such as sorting, distribution, and interpolation.
It improves the programmability and data routing capabilities of the rasterization stage, significantly reduces memory access and scheduling overhead, improves hardware throughput and rendering performance, and achieves end-to-end frame rate improvement.
Smart Images
Figure CN121982180B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer graphics technology, specifically relating to a global neural rendering method and system based on a programmable rasterization engine. Background Technology
[0002] In recent years, neural rendering has gradually evolved from the traditional graphics pipeline centered on meshes to a hybrid paradigm that integrates explicit geometry and learned representations. Explicit neural representations, represented by 3D Gaussian Splashing (3D-GS), have become an important approach due to their training / rendering efficiency and cross-view consistency. However, their runtime data flow and operator types differ significantly from traditional mesh rasterization, making it difficult for the two types of workloads to coexist efficiently under unified hardware and scheduling.
[0003] Looking at the stage-by-stage overhead of a typical 3D-GS rendering pipeline, rasterization and pixel-level rendering / compositing together account for approximately 90% of the total time, while the preprocessing stage accounts for only a small percentage. This distribution remains relatively stable across different scenes and resolutions, indicating that the bottleneck is stage-specific. Further breakdown shows that pixel rendering relies more on FP32 computing power, while the rasterization stage is more sensitive to video memory bandwidth. These two stages constitute a typical dual bottleneck of computing power / bandwidth separation.
[0004] To support mesh and Gaussian blending rendering on a unified platform, existing research proposes a generalized three-stage rendering / rasterization pipeline for AI accelerators: object processing, screen processing, and pixel processing. Among them, the screen processing stage is responsible for generalized rasterization and screen space scheduling (such as bucketing, routing, and reordering), which are mostly fixed functions in traditional GPUs, but AI accelerators often lack corresponding dedicated units and need to be mapped in a programmable way.
[0005] Tile rendering (TBR) is widely adopted to align with the on-chip sticky memory (SPM) and in-core parallelism of AI accelerators: the rasterization and shading of the same batch of primitives can be completed within a single AI core, with intermediate results residing in the SPM, reducing multiple main memory round trips. At the same time, it makes the bidirectional mapping of "object → tile" and "tile → primitive" explicit, facilitating data routing and rearrangement in screen space.
[0006] Although tile rendering alleviates memory access pressure, it still incurs several types of overhead due to the high overlap characteristics of 3D-GS and the edge function / interpolation requirements of the mesh:
[0007] (1) Global sorting and repetition: In 3D-GS, it is often necessary to repeat and sort primitive-Tile pairs over a large range. The sorting and repetition sub-stages have been tested as bottlenecks.
[0008] (2) Excessive Gaussian-Tile Interaction: The number of tiles covered by each Gaussian on average is much greater than the number of Gaussians contained in a single tile, resulting in increased invalid interactions and memory access;
[0009] (3) Side effects of large tiles: Large tile size will significantly increase the total number of 3D-GS primitives in the rendering, and increase the cost of sorting and deduplication.
[0010] Therefore, existing implementations emphasize the importance of multi-stage pruning / elimination, especially early pruning, which can effectively reduce the amount of invalid work in memory-constrained sub-stages and improve overall resource utilization. At the same time, they attempt to break down the global operations of "creating tiles / repeating / key sorting" into earlier stages and complete them at the tile level to alleviate global interference.
[0011] On the other hand, AI accelerators differ from traditional GPUs in their capability configuration. In AI accelerators, some operations in generalized rasterization are only "weakly supported" or "not supported," requiring the concatenation of multiple general instructions, leading to instruction sequence bloat and decreased scheduling efficiency. This directly impacts the performance and energy efficiency of critical steps such as edge function computation, interpolation, prefix scan, and Z-cull.
[0012] It should be noted that while volume rendering paths like NeRF can bypass fixed-function raster units in screen processing to some extent, screen space scheduling remains an indispensable core component in unified engines oriented towards hybrid representations (mesh + Gaussian). This further emphasizes the necessity of providing programmable screen processing and rasterization capabilities on AI accelerators. Summary of the Invention
[0013] In view of the above, the purpose of this invention is to provide a global neural rendering method and system based on a programmable rasterization engine. By mapping the programmable rasterization engine to the vector kernel of an AI accelerator, the neural rendering workload of multiple representations such as mesh and 3D Gaussian splashing can be uniformly supported on the AI accelerator, thereby improving the programmability and data routing capability of the rasterization stage. Furthermore, by using direct memory access unit instructions and new rasterization instructions, the memory access and scheduling overhead of key steps such as sorting / distribution / interpolation can be reduced. This achieves a systematic improvement in end-to-end frame rate and non-neural module performance while maintaining extremely low hardware increment.
[0014] To achieve the above-mentioned objectives, the present invention provides the following technical solution:
[0015] In a first aspect, embodiments of the present invention provide a global neural rendering method based on a programmable rasterization engine, comprising the following steps:
[0016] At the programmable rasterization engine deployed within the vector core of the AI accelerator, the rasterization descriptor is parsed according to the rasterization instructions issued by the control core in order to extract vector micro-operations and control parameters.
[0017] The task state machine is maintained according to the control parameters and control signals are distributed. The execution entry point is selected from the preset vector kernel table according to the control signals, and the vector micro-operations are instantiated into parallel vector threads.
[0018] During the execution of the vector thread, the data dependencies of the vector registers and the synchronization status with the direct memory access unit are tracked. Based on this, vector load / store operations are performed to move data between the registers and the on-chip shared memory, and ready vector micro-operations are dynamically scheduled to the execution unit to complete the unified rasterization calculation of the multi-primitive representation.
[0019] The rasterization calculation results are output to the neural rendering network to complete the global neural rendering.
[0020] Preferably, the rasterization instructions include a first rasterization acceleration instruction and a second rasterization acceleration instruction, which are used to trigger the execution of side function calculation and to trigger the execution of pixel interpolation or synthesis, respectively.
[0021] Preferably, the control parameters include: function code, task type, number of primitives, batch size, tile parameters, and source / target address and data step size in on-chip shared memory, wherein the function code is used to select different side function calculation modes, pixel interpolation modes, or synthesis modes.
[0022] Preferably, the vector kernel table is programmable to support rasterization computation of a variety of geometric primitives, including at least mesh triangles and 3D Gaussian splashes.
[0023] Preferably, during the execution of the vector thread, vector loading and storage are performed according to the step size in the control parameters to move data in batches, and arithmetic logic operations are performed using single instruction multiple data vector operations.
[0024] Preferably, the vector micro-operation sequences executing the first rasterization acceleration instruction and the second rasterization acceleration instruction share a double buffer in the on-chip shared memory, and the two types of operations are executed overlappingly between different tiles or batches.
[0025] Preferably, the programmable rasterization engine is configured to determine its set of reachable tiles based on the screen extent of each primitive, and to perform rasterization processing tasks only on the tiles in the set of reachable tiles.
[0026] Preferably, the direct memory access unit is configured to distribute the common features of primitives to the buffers of multiple corresponding on-chip shared memory in a single read and multi-target write manner according to the mapping relationship between tiles and primitives, so that the common features can be reused among multiple tiles.
[0027] Preferably, the AI accelerator includes at least:
[0028] The control core is used to schedule the rasterization pipeline, generate rasterization instructions, and issue corresponding rasterization descriptors.
[0029] The vector kernel, which includes the programmable rasterization engine, is used to execute rasterization instructions issued by the control kernel;
[0030] On-chip shared memory is used to store primitive data, tile lists, edge function coefficients, and tile-level pixel intermediate results;
[0031] The Direct Memory Access Unit (DMI) is used to move data between off-chip memory and on-chip shared memory, and supports the distribution of single primitive data to storage areas corresponding to multiple screen tiles.
[0032] Secondly, embodiments of the present invention also provide a global neural rendering system based on a programmable rasterization engine, used to implement the above-mentioned global neural rendering method based on a programmable rasterization engine, characterized in that it includes: an instruction parsing and parameter extraction module, a kernel control and thread instantiation module, a synchronization tracking and operation scheduling module, and a result output and neural rendering module.
[0033] The instruction parsing and parameter extraction module is used at the programmable rasterization engine deployed in the vector core of the AI accelerator to parse the rasterization descriptor according to the rasterization instruction issued by the control core, so as to extract vector micro-operations and control parameters.
[0034] The kernel control and thread instantiation module is used to maintain the task state machine according to the control parameters and distribute control signals. Based on the control signals, it selects the execution entry point from the preset vector kernel table and instantiates the vector micro-operations into parallel vector threads.
[0035] The synchronization tracking and operation scheduling module is used to track the data dependencies of the vector register and the synchronization status with the direct memory access unit during the execution of the vector thread, and accordingly execute vector loading / storage operations to move data between the register and the on-chip shared memory, and dynamically schedule ready vector micro-operations to the execution unit to complete the unified rasterization calculation of the multi-primitive representation.
[0036] The result output and neural rendering module is used to output the rasterization calculation results to the neural rendering network to complete global neural rendering.
[0037] Compared with the prior art, the beneficial effects of the present invention include at least the following:
[0038] (1) By introducing programmable rasterization instructions and the corresponding vector kernel table, the present invention enables the same hardware to flexibly adapt to primitive calculations of different representations such as mesh and 3D Gaussian splash, unifies the rendering interface, and improves the programmability and flexibility of the rasterization stage.
[0039] (2) The present invention adopts tile-based rendering collaboration, combined with the multicast distribution function of the direct memory access unit, so that primitive data can be read once on the chip and reused by multiple blocks, which greatly reduces the frequency of off-chip memory access and bandwidth occupation, and significantly reduces the memory access and scheduling overhead of key steps.
[0040] (3) This invention uses the dual buffer, scoring board and synchronization mechanism of on-chip shared memory to perform fine-grained dependency management of data transport and calculation stages, realizes overlapping execution between different blocks or batches, and enables common features to be reused between multiple tiles between direct memory access units and on-chip shared memory, effectively reducing memory access latency, improving hardware throughput and rendering performance, and realizing efficient pipeline parallelism and resource utilization. Attached Figure Description
[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0042] Figure 1 This is a flowchart illustrating the global neural rendering method based on a programmable rasterization engine provided in an embodiment of the present invention.
[0043] Figure 2 This invention provides a modular implementation process and data / control relationship on an AI accelerator.
[0044] Figure 3 This is a schematic diagram of the control component structure of the vector kernel based on the programmable rasterization engine provided in an embodiment of the present invention;
[0045] Figure 4 This is a schematic diagram of the first rasterization acceleration instruction - edge computing provided in an embodiment of the present invention;
[0046] Figure 5 This is a schematic diagram of the second rasterization acceleration instruction-interpolation calculation provided in an embodiment of the present invention;
[0047] Figure 6This is a schematic diagram of the vector kernel execution flow of the first rasterization acceleration instruction-edge computing provided in an embodiment of the present invention;
[0048] Figure 7 This is a schematic diagram of the structure of a global neural rendering system based on a programmable rasterization engine provided in an embodiment of the present invention. Detailed Implementation
[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.
[0050] The inventive concept of this invention is as follows: Addressing the shortcomings of existing AI accelerators in providing unified and efficient support for multi-representation neural rendering tasks such as mesh and 3D Gaussian splashing, as well as the weak programmability, low data reuse rate, and high memory access and scheduling overhead during the rasterization stage, this invention provides a global neural rendering method and system based on a programmable rasterization engine. By introducing a programmable rasterization engine driven by rasterization instructions into the vector kernel, and coordinating it with the control kernel, on-chip shared memory, and direct memory access unit with multicast capabilities, a complete hardware acceleration system is constructed, encompassing instruction parsing, task scheduling, intelligent data distribution, and parallel collaborative execution of the computational pipeline. This achieves a reduction in critical memory access overhead and a comprehensive improvement in computational efficiency.
[0051] like Figure 1 As shown, this embodiment of the invention provides a global neural rendering method based on a programmable rasterization engine, including the following steps:
[0052] S1, at the programmable rasterization engine deployed in the vector core of the AI accelerator, parses the rasterization descriptor according to the rasterization instructions issued by the control core in order to extract vector micro-operations and control parameters;
[0053] S2, maintains the task state machine according to the control parameters and distributes control signals, selects the execution entry point from the preset vector kernel table according to the control signals, and instantiates the vector micro-operations into parallel vector threads;
[0054] S3, during the execution of the vector thread, tracks the data dependencies of the vector registers and the synchronization state with the direct memory access unit, and performs vector load / store operations accordingly to move data between the registers and the on-chip shared memory, and dynamically schedules ready vector micro-operations to the execution unit to complete the unified rasterization calculation of the multi-primitive representation;
[0055] S4 outputs the rasterization calculation results to the neural rendering network to complete the global neural rendering.
[0056] like Figure 2 As shown, the programmable rasterization engine (F) provided in the embodiment 2 IRE operates in the screen space processing stage of the three-stage pipeline parallel collaborative processing: (1) Object Processing: Reads view and object data, including frustum clipping, invalid primitive removal, projection / screen space transformation, necessary (depth / importance) sorting, feature calculation and binning judgment for mesh triangles and Gaussian primitives, performs Setup on mesh triangles to obtain the primitive parameters and attribute primitives required for subsequent edge function calculation, and generates primitive packets and their screen extents; (2) Screen Processing: Divides the screen into multiple tiles, calculates the reachable tile set based on the primitive screen extents. The system establishes a bidirectional mapping of "Tile → Primitive List / Primitive → Tile List" and completes binning of primitives, reordering, feature reading, object-by-object distribution, feature culling, and feature organization. Ctl-Core issues tasks only to the relevant tiles / cores based on the reached tile set, avoiding invalid interactions on full-screen tiles. Within each tile, the first rasterization acceleration instruction raster.edge is executed to calculate the edge function coefficients and prepare for coverage testing and optional early Z clipping. Then, the second rasterization acceleration instruction raster.interp is executed to interpolate the attributes of pixels / fragments that have passed the coverage test, and selects the corresponding composition mode according to the function code to complete Alpha composition or accumulation. Finally, the interpolated pixel attributes or intermediate features are written back to SPM. The above intermediate data resides on the on-chip SPM first, and primitive packets, bucket lists and common features are written into the SPM; for primitives that reach multiple tiles, common features are distributed to the SPM buffers corresponding to multiple tiles in a multicast manner to reduce the round-trip access of dynamic random access memory (DRAM) and improve pipeline throughput; (3) Pixel Processing: The features calculated by screen space processing are sent into the pre-trained neural network for inference calculation; the feature activation values pass through the feature-oriented global illumination neural network, output neural network and post-processing neural network in sequence to generate the final image (inference result).
[0057] F 2IRE is designed for converged graphics rendering and AI inference workloads, deployed on AI processors / accelerators that include a control core (Ctl-Core), one or more vector cores (V-Cores), on-chip shared memory (SPM), direct memory access (DMA) units, and on-chip interconnects / on-chip networks (optionally including matrix cores (M-Cores) for neural network inference). Specifically, the Ctl-Core generates rendering tasks and generates / decodes rasterization descriptors; the V-Cores execute programmable rasterization microkernels such as side functions and interpolation; the SPM stores primitive data, tile bucket lists, side function coefficients, and tile-level intermediate results; and the DMA moves and distributes data between DRAM and the SPM, and between different tile-corresponding SPM regions. The Ctl-Core triggers DMA data fetching / distribution and V-Core computation by writing descriptors and issuing instructions. The V-Core fetches data from the SPM and writes back the results after dependencies are satisfied, thus forming a pipelined execution of "fetch-compute-write-back".
[0058] like Figure 3 As shown, by F 2The IRE is mapped to the V-Core, and its control components include: a descriptor resolver, a vector kernel control unit, a vector operation management unit, a vector thread control unit, a vector register scoreboard, a vector memory access (vector loading / vector storage) unit, and a vector operation scheduling unit. The descriptor resolver parses the rasterization descriptors from the Ra / Rb / Rc registers written by the Ctl-Core and writes them to the task register. The parsed fields include at least: (a) control fields: function code / task type / composition mode; (b) scale fields: number of primitives and batch size; (c) geometry and parallelism fields: maximum X / Y coordinates of the tile, tile size, and mask per tile or its generation method; (d) data flow fields: sticky note memory source / destination address and stride (Src / Dst Addr, Stride). After the Ctl-Core writes the descriptor, it issues a raster.edge or raster.interp instruction. The V-Core parses the descriptor and accesses the SPM accordingly to complete batch execution. The vector kernel control unit selects the corresponding first rasterization acceleration instruction `raster.edge` or the second rasterization acceleration instruction `raster.interp` microkernel entry point based on the function code. The vector operation management unit breaks down the microkernel into vector arithmetic / logic / special functions, selects the execution entry point corresponding to different computation modes from the vector kernel table, and schedules its execution. The vector thread control unit instantiates vector micro-operations into multiple parallel vector threads based on the batch size. The vector register scoreboard tracks and manages the data dependencies between vector registers and the synchronization status with the DMA unit during vector thread execution. The vector memory access unit performs batch loading and writing back of the SPM according to address and step size. The vector operation scheduling unit dynamically schedules vector micro-operations to the corresponding vector execution units for execution based on the data status indicated by the vector register scoreboard.
[0059] In terms of module collaboration, Ctl-Core first configures descriptors based on the reachable tile set and bucketing results generated in the object / screen phase and triggers DMA data fetching / distribution. The vector register scoreboard records the DMA task and its target buffer. Only after the DMA transfer / distribution is completed and an acknowledgment signal is generated does V-Core allow the issuance of raster.edge or raster.interp commands to the corresponding buffer. The edge function coefficients and auxiliary quantities written back in the raster.edge phase are read and used in the raster.interp phase after the vector register scoreboard is marked as ready, thus ensuring... Figure 2 The "fetch data-calculate-write back" process in the pipeline shown can be executed correctly and overlappingly according to dependencies.
[0060] To match the memory access and parallelism requirements of graphics-AI fusion workloads, end-to-end rendering acceleration for multiple primitives in mesh computing is achieved by providing programmable rasterization instructions and on-chip data residency mechanisms at the vector kernel level. Figure 4 and Figure 5 As shown, this invention introduces a first rasterization acceleration instruction, raster.edge, and a second rasterization acceleration instruction, raster.interp, into the AI accelerator. raster.edge is used for batch execution of edge function coefficient calculation, coverage test preparation, and optional early Z-clipping during the Setup / Edge phase; raster.interp is used for batch execution of pixel / fragment attribute interpolation within a tile, optional weight / depth updates, and write-back / compositing. Both instructions are configured by Ctl-Core using rasterization descriptors encoded with three registers (Ra / Rb / Rc). The descriptor of the first rasterization acceleration instruction, raster.edge, includes at least the following fields: function code, task type, number of primitives, batch size, tile parameters (maximum X coordinate, maximum Y coordinate, block size), note memory address for each tile mask, note memory source address for primitive features, note memory target address for primitive feature calculation results, and note memory source address stride for primitive features. The descriptor of the second rasterization acceleration instruction, raster.interp, includes at least the following fields: function code, number of interpolated features, number of primitives, batch size, tile parameters (maximum X coordinate, maximum Y coordinate, size), note memory address for edge calculation results, note memory source address for primitive vertices, note memory target address for primitive interpolation results, and data distribution format of primitive features.
[0061] For details on the microkernel execution flow of raster.edge, please refer to [link / reference]. Figure 6 ① Batch load primitive parameters → ② Issue a primitive data loading request required for edge computing → ③ Load primitive data from SPM → ④ Loading primitive data required for edge computing is complete → ⑤ Calculate edge function coefficients and auxiliary quantities → ⑥ Issue a request to store the results of edge computing → ⑦ Store and write the edge computing results back to SPM → ⑧ SPM returns a result writing completion signal to the vector storage unit → ⑨ Generate a completion flag to ensure that the interpolation stage can start after the dependencies are satisfied.
[0062] Furthermore, the output buffer (edge function coefficients / auxiliary quantities) of raster.edge and the input buffer of raster.interp correspond one-to-one in the descriptor, and the data dependency of "edge first, interp second" is established through the vector register scoreboard; the raster.interp stage can write the interpolated pixel attributes to the Frame Buffer, or to the input feature buffer of the neural rendering / relighting network, so as to seamlessly connect with the neural network inference stage on the matrix kernel M-Core on the same AI accelerator.
[0063] In terms of tile processing collaboration, F 2 IRE issues tasks only to the relevant tiles / cores based on the reached tile set, avoiding invalid interactions across the entire screen tiles. The screen is divided into multiple tiles. The tile-primal list and common primitive features generated during the screen processing stage can be written to the SPM buffers corresponding to multiple tiles via DMA multicast descriptors (DMA.distribution) in a multicast / distribution manner, so that the same primitive / feature only needs to be read from DRAM once and can be reused by multiple tiles. V-Core completes bucket list reading, raster.edge edge function evaluation, raster.interp attribute interpolation, and alpha synthesis within the tile. Intermediate results reside in the SPM as much as possible and are only written back to DRAM / Frame Buffer when necessary to reduce memory access overhead.
[0064] In the 3D-GS path, global sorting can be decomposed into "early depth sorting + tile-level local sorting", and different sorting and composition strategies are used for transparent and opaque objects. In the mesh path, the triangle setup results and Gaussian projection primitives enter the screen processing and pixel processing stages using a unified primitive data format, thereby realizing multi-primitive isomorphic access and a unified raster interface.
[0065] Furthermore, the DMA unit supports at least two types of data operations for TBR: (i) hybrid fetching, used to fetch data blocks of different layouts and write them to SPM according to index / address generation rules; (ii) distribution / multicasting, used to distribute the common features of the same primitive to the SPM buffers corresponding to multiple tiles in a single read and multi-target write manner according to the Tile→primitive mapping, and can be combined with a multicast lookup table and a receive counter to support the synchronization of distribution completion.
[0066] Furthermore, V-Core internally sets up a vector kernel table, where function codes in the descriptors are used to select different edge function calculation modes, interpolation modes, and composition modes. V-Core loads / writes back data in batches by step size through vector loading / vector storage, and uses vector operations to perform subtraction, multiplication, multiply-add, and reciprocal approximation operations to support different primitive types and their variant extensions.
[0067] Furthermore, to overlap the "DMA fetch / distribution - edge function calculation - interpolation calculation - write-back / composition" process, Ctl-Core configures at least one set of dual buffers in SPM and manages the dependencies of each stage through a scoreboard and a fence / identifier. When the current batch is being calculated in one set of buffers, the next batch can be prefetched / distributed to another set of buffers by DMA. The buffer is switched when the fence / identifier indicates that the preceding stage is completed. This allows the "DMA fetch / distribution" and "edge function / interpolation calculation" to be executed overlappingly between different tiles / batches, thereby reducing pipeline cavitation and improving end-to-end throughput.
[0068] In addition, the following optional and variant schemes are further supported:
[0069] Cross-platform porting: The three-stage pipeline and descriptor abstraction of this invention can be reused on other general AI accelerators with SIMD (Single Instruction Multiple Data) vector cores and on-chip SPM. Only the descriptor parsing, DMA distribution / multicast and Fence / Token synchronization mechanisms need to be adapted.
[0070] Primitive extension: Without changing the overall framework, function codes and microkernel tables can be extended to support more primitive types (such as line segments, point sprites, etc.) and different interpolation / composition modes;
[0071] Precision extension: Precision / quantization configuration fields can be added to the descriptor to support the storage and calculation of low-precision attributes such as BF16 / FP16 / FP8, further reducing bandwidth and power consumption.
[0072] In summary, the global neural rendering method based on a programmable rasterization engine provided by this invention has been experimentally verified to achieve end-to-end frame rate improvement under the programmable rasterization engine: In 3D-GS tasks, based on the pipeline and engine of this invention, a frame rate improvement of approximately 21% to 39% is achieved on platforms such as the RTX 4090; among which, pipeline optimization alone can achieve an overall speed improvement of 15% to 29%, and the engine further brings an additional gain of 6% to 10%. Gain in non-neural components of the fusion scene: In the Gaussian-mesh fusion pipeline, non-neural network components can achieve a performance improvement of approximately 6% to 12% (platform coverage H100 / 4090 / 3090). Extremely low hardware overhead: While maintaining programmable functionality and flexible routing, the hardware area / overhead increase is within approximately 2%; high pixel interpolation throughput can be achieved, and it has good scalability. Uniformity and scalability: The engine abstracts the requirements of multi-primitive raster with unified instructions and descriptors, which facilitates future expansion to more scene representations and low-precision rendering (BF16 / FP8).
[0073] Based on the same inventive concept, such as Figure 7 As shown, this embodiment of the invention also provides a global neural rendering system 700 based on a programmable rasterization engine, including: an instruction parsing and parameter extraction module 710, a kernel control and thread instantiation module 720, a synchronization tracking and operation scheduling module 730, and a result output and neural rendering module 740.
[0074] The instruction parsing and parameter extraction module 710 is used to parse the rasterization descriptor according to the rasterization instructions issued by the control core at the programmable rasterization engine deployed in the vector core of the AI accelerator, so as to extract vector micro-operations and control parameters.
[0075] The kernel control and thread instantiation module 720 is used to maintain the task state machine according to the control parameters and distribute control signals. Based on the control signals, it selects the execution entry point from the preset vector kernel table and instantiates the vector micro-operations into parallel vector threads.
[0076] The synchronization tracking and operation scheduling module 730 is used to track the data dependencies of vector registers and the synchronization status with direct memory access units during the execution of vector threads. Based on this, it performs vector load / store operations to move data between registers and on-chip shared memory, and dynamically schedules ready vector micro-operations to the execution unit to complete the unified rasterization calculation of multi-primitive representations.
[0077] The result output and neural rendering module 740 is used to output the rasterization calculation results to the neural rendering network to complete the global neural rendering.
[0078] The specific implementation process of this system can be found in the embodiment of the global neural rendering method based on the programmable rasterization engine, which will not be repeated here.
[0079] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A global neural rendering method based on a programmable rasterization engine, characterized in that, Includes the following steps: At the programmable rasterization engine deployed within the vector core of the AI accelerator, the rasterization descriptor is parsed according to the rasterization instructions issued by the control core in order to extract vector micro-operations and control parameters. The task state machine is maintained according to the control parameters and control signals are distributed. The execution entry point is selected from the preset vector kernel table according to the control signals, and the vector micro-operations are instantiated into parallel vector threads. During the execution of the vector thread, the data dependencies of the vector registers and the synchronization status with the direct memory access unit are tracked. Based on this, vector load / store operations are performed to move data between the registers and the on-chip shared memory, and ready vector micro-operations are dynamically scheduled to the execution unit to complete the unified rasterization calculation of the multi-primitive representation. The rasterization calculation results are output to the neural rendering network to complete the global neural rendering.
2. The global neural rendering method based on a programmable rasterization engine according to claim 1, characterized in that, The rasterization instructions include a first rasterization acceleration instruction and a second rasterization acceleration instruction, which are used to trigger the execution of side function calculation and to trigger the execution of pixel interpolation or synthesis, respectively.
3. The global neural rendering method based on a programmable rasterization engine according to claim 1, characterized in that, The control parameters include: function code, task type, number of primitives, batch size, tile parameters, and source / target address and data step size in on-chip shared memory, wherein the function code is used to select different side function calculation modes, pixel interpolation modes, or synthesis modes.
4. The global neural rendering method based on a programmable rasterization engine according to claim 1, characterized in that, The vector kernel table is programmable and configurable to support rasterization computation of a variety of geometric primitives, including at least mesh triangles and 3D Gaussian splashes.
5. The global neural rendering method based on a programmable rasterization engine according to claim 1 or 3, characterized in that, During the execution of the vector thread, vector loading and storage are performed according to the step size in the control parameters to move data in batches, and arithmetic logic operations are performed using single instruction multiple data vector operations.
6. The global neural rendering method based on a programmable rasterization engine according to claim 2, characterized in that, The vector micro-operation sequences executing the first rasterization acceleration instruction and the second rasterization acceleration instruction share a double buffer in the on-chip shared memory, and enable the two types of operations to be executed overlappingly between different tiles or batches.
7. The global neural rendering method based on a programmable rasterization engine according to claim 1, characterized in that, The programmable rasterization engine is configured to determine its set of reachable tiles based on the screen extent of each primitive, and to perform rasterization processing tasks only on the tiles in the set of reachable tiles.
8. The global neural rendering method based on a programmable rasterization engine according to claim 1, characterized in that, The direct memory access unit is configured to distribute the common features of primitives to the buffers of multiple corresponding on-chip shared memory in a single read and multi-target write manner according to the mapping relationship between tiles and primitives, so that the common features can be reused among multiple tiles.
9. The global neural rendering method based on a programmable rasterization engine according to claim 1, characterized in that, The AI accelerator includes at least: The control core is used to schedule the rasterization pipeline, generate rasterization instructions, and issue corresponding rasterization descriptors. The vector kernel, which includes the programmable rasterization engine, is used to execute rasterization instructions issued by the control kernel; On-chip shared memory is used to store primitive data, tile lists, edge function coefficients, and tile-level pixel intermediate results; The Direct Memory Access Unit (DMI) is used to move data between off-chip memory and on-chip shared memory, and supports the distribution of single primitive data to storage areas corresponding to multiple screen tiles.
10. A global neural rendering system based on a programmable rasterization engine, used to implement the global neural rendering method based on a programmable rasterization engine as described in any one of claims 1 to 9, characterized in that, include: Instruction parsing and parameter extraction module, kernel control and thread instantiation module, synchronization tracing and operation scheduling module, result output and neural rendering module; The instruction parsing and parameter extraction module is used at the programmable rasterization engine deployed in the vector core of the AI accelerator to parse the rasterization descriptor according to the rasterization instruction issued by the control core, so as to extract vector micro-operations and control parameters. The kernel control and thread instantiation module is used to maintain the task state machine according to the control parameters and distribute control signals. Based on the control signals, it selects the execution entry point from the preset vector kernel table and instantiates the vector micro-operations into parallel vector threads. The synchronization tracking and operation scheduling module is used to track the data dependencies of the vector register and the synchronization status with the direct memory access unit during the execution of the vector thread, and accordingly execute vector loading / storage operations to move data between the register and the on-chip shared memory, and dynamically schedule ready vector micro-operations to the execution unit to complete the unified rasterization calculation of the multi-primitive representation. The result output and neural rendering module is used to output the rasterization calculation results to the neural rendering network to complete global neural rendering.