Nerve radiation field rendering acceleration method and device

Abstract

The invention belongs to the technical field of neural radiation field (NeRF) rendering acceleration, particularly relates to a neural radiation field rendering acceleration method and device, aims at ASIC (Application Specific Integrated Circuit) rendering acceleration of an MERF algorithm, provides an algorithm-hardware collaborative design framework aiming at the problem of large overhead of off-chip memory access and on-chip memory in a boundless scene, and comprises the following steps of: 1, optimizing a ray information storage mode, ray key parameters are used for replacing sampling point storage, so that on-chip storage is greatly reduced; 2, a ray synchronous propulsion process: sequentially rendering according to voxels closest to a camera, and reducing repeated loading of macro voxels; 3, realizing parallelism of voxel traversal and space jump through a master-slave ray table structure; 4, streamlined chip architecture is carried out, and processes of ray initialization, rendering, delayed coloring and the like are completed; 5, performing quadrant compression adaptation, and processing cross-quadrant ray refraction and scene penetration through a voxelid flag bit; 6, combining a plurality of voxels into macro voxels to reduce repeated access of a single voxel; 7, carrying out jump grid compression, and reducing the memory consumption of a grid bitmap to 50KB; and finally realizing a boundary-free scene real-time rendering effect.

Description

A method and apparatus for accelerating neural radiation field rendering Technical Field

[0001] This invention relates to the field of Neural Radiation Field (NeRF) rendering acceleration technology, specifically to a Neural Radiation Field rendering acceleration method and apparatus. Background Technology

[0002] Neural Radiance Fields (NeRF) is a groundbreaking technology in the field of 3D vision and graphics in recent years. It learns to encode the spatial position (x, y, z) and viewing direction (θ, φ) of a scene into continuous 5D functions through a multilayer perceptron (MLP). It can learn how to accurately represent the geometry, appearance and lighting of a scene, and synthesize realistic new perspective images from sparse input images. It is widely used in fields such as new perspective synthesis, virtual reality and digital twins.

[0003] However, neural radiance fields and their variant, Memory-Efficient Radiance Fields (MERF), require extensive sampling of each ray emitted by the camera in 3D space during rendering. They also need to query neural networks to obtain color and density, and finally calculate pixel color through volumetric rendering integration. This process is extremely computationally intensive, resulting in slow rendering speeds that fail to meet the demands of real-time interactive applications, severely hindering its practical application and widespread adoption.

[0004] To address this, the industry has proposed various rendering acceleration solutions, mainly categorized into algorithm optimization and hardware acceleration. At the algorithm optimization level, existing technologies primarily reduce computational overhead by optimizing sampling strategies or rendering processes. For example, Chinese patent CN117911604A discloses a sampling optimization-based method that differentiates sampling rays by detecting the foreground and background of the image and analyzing texture complexity, dynamically adjusting the number of sampling points based on resolution during rendering. While this method reduces unnecessary sampling, it is essentially a software optimization based on general-purpose processors, and its acceleration effect is limited for the fundamental problem of massive rays and sampling points in boundless scenes. Another approach, as shown in Chinese patent CN114581581A, adaptively divides the target rendering image into several region blocks and performs volume rendering for each region block, aiming to allocate computational resources more efficiently in two-dimensional space. However, this method focuses on the division and composition of the rendering image and does not address the core memory access bottleneck issue in three-dimensional volume rendering.

[0005] At the hardware acceleration level, to address the high cost of neural radiation field rendering, a significant amount of research has focused on dedicated accelerators for algorithms. Traditional accelerator designs are typically non-end-to-end: one approach is to pre-compute and store all ray sampling points off-chip, then load sampling points within the same voxel onto the chip for processing during rendering; another approach is to load rays off-chip, render rays within the same coarse-grained voxel (composed of multiple voxels), and then return the rays off-chip after they pass through the coarse voxel or rendering is complete. Although these approaches aim to reduce the repetitive loading of voxel data, in unbounded scenes, the number of rays and sampling points increases exponentially. Frequent transfer of ray or sampling point data between off-chip and on-chip introduces massive amounts of non-voxel off-chip memory access, becoming a major bottleneck limiting performance.

[0006] If an end-to-end hardware rendering workflow is adopted, that is, the processing of the entire ray is completed on the chip, the aforementioned off-chip memory access can be avoided, but new challenges will be introduced: if all the sampling point information on a ray is pre-calculated and stored on the chip, it will consume huge amounts of on-chip storage resources, severely limiting the number of rays that can be processed in parallel on the chip, resulting in low rendering throughput; at the same time, the limited on-chip capacity also forces the system to frequently switch rendering regions, thereby causing repeated loading of voxel data, which also makes it impossible to achieve efficient real-time rendering.

[0007] Existing neural radiation field acceleration technologies, whether through software algorithm optimization or traditional hardware architecture, have significant shortcomings when facing the real-time rendering requirements of unbounded, large-scale scenes. Therefore, there is an urgent need in this field for a dedicated hardware acceleration method and device for neural radiation fields that can optimize memory access, significantly reduce on-chip storage overhead, and achieve efficient parallel computing, in order to systematically solve the above problems. Summary of the Invention

[0008] In view of this, the present invention aims to solve the problem that existing neural radiation field acceleration technologies cannot fundamentally solve the computation and memory access bottlenecks on the software side, while on the hardware side they are caught in the dilemma of "off-chip memory access overhead" and "on-chip storage limitations", and provides a neural radiation field rendering acceleration method and device.

[0009] A method for accelerating neural radiation field rendering includes: replacing the traditional sampling point storage method with a ray information table, and executing a ray synchronization algorithm based on the ray information table, so that the same voxel can be shared and accessed by all related rays in one processing, thereby significantly reducing off-chip memory access; at the same time, by introducing a master-slave ray information table structure, parallel execution of ray traversal and spatial jump is realized to hide computational latency and improve overall rendering efficiency.

[0010] Specifically, this includes: Step 1: Storing parameterized information of ray information.

[0011] To address the issue of traditional end-to-end rendering methods that calculate and store sampling points on a ray on-chip, resulting in significant on-chip storage overhead, this invention employs a more convenient ray information storage method. Instead of storing all sampling points, only a simplified parameter of a single ray is stored on-chip, which can significantly reduce storage overhead.

[0012] An on-chip ray information table is maintained. Each entry in the ray information table stores parameterized information of a single ray, including: the pixel identifier (pixel_id) corresponding to the ray, the origin coordinates (origin), the direction vector (direction), the ray travel distance (distance), the identifier of the current voxel (voxel_id), and the number of steps (times) within the voxel.

[0013] Preferably, by storing the above-mentioned fixed-size ray parameters, rather than the complete information of all sampling points on the ray, on-chip storage consumption can be reduced by orders of magnitude.

[0014] 2. Execute the ray synchronization propulsion algorithm.

[0015] The rendering process is performed in blocks. For the current rendering block, the following iterative steps are executed: a. Voxel traversal: Traverse the ray information table and find the voxel_id closest to the camera's starting point based on the voxel identifier voxel_id of all ray entries.

[0016] b. Ray Collection and Submission: Traverse the ray information table again, collect all rays located in the nearest voxel found in step a, and input the corresponding ray information entries into the rendering FIFO.

[0017] c. Parallel rendering and spatial jump: The sub-block rendering unit actively retrieves ray segments from the FIFO, performs step sampling, interpolation calculation, feature vector accumulation, and volume rendering, and stores the results in the intermediate value buffer; while retrieving ray segments from the FIFO, the sub-block rendering unit inputs the ray segments into the spatial jump unit, calculates the next non-empty voxel through the ray-voxel intersection algorithm, and updates the ray travel distance, voxel identifier (voxel_id), and step count (times) in the ray information table.

[0018] d. Looping and Termination: After completing steps b and c, the voxel traversal module begins the next round of nearest voxel lookup based on the updated ray information table. This process repeats until the voxel traversal module finds that all rays have been rendered and their voxel_id is marked as complete.

[0019] Furthermore, to address the issue of pipeline blockage caused by waiting for all rays to complete spatial jumps during voxel traversal in synchronous propagation, this invention employs a master-slave ray information table structure to shorten and reduce computational latency: Master ray information table: responsible for recording the voxel information of the current location of the ray, for use by the voxel traversal module; Slave ray information table: responsible for recording the next voxel information that the ray will enter after spatial jump, and when the ray information in the master table is taken, the slave table fills the master table.

[0020] The structure realizes the pipeline parallelism of "voxel traversal-ray rendering-space jump", which greatly improves the hardware utilization. Its workflow is as follows: (1) The voxel traversal module traverses the main table, finds the target voxel, and then notifies the ray traversal module.

[0021] (2) The ray traversal module collects rays within the target voxel from the main table, sends them into the rendering pipeline, and immediately fills the empty positions in the main table with the ray entries from the corresponding positions in the table.

[0022] (3) The ray entry sent into the rendering pipeline has its spatial jump module write the calculation results (next voxel information) back to the table.

[0023] Furthermore, entries in the ray information table can be marked as pending; when a ray entry that needs to be filled in step c is in a pending state, ray traversal is blocked, ray collection is paused, and the update of the entry is waited for.

[0024] Furthermore, a flag bit is introduced to address the refraction problem of rays crossing quadrants caused by quadrant compression in the MERF algorithm, ensuring correct update and synchronization when rays cross quadrants; the high byte of the voxel identifier voxel_id is configured as a flag bit.

[0025] Furthermore, the flag bits include: ① normal rendering flag (e.g., 0); ② entering a new quadrant flag (e.g., 1); ③ ray rendering completed or scene penetration flag (e.g., 2); ④ ray is in a pending state (e.g., 3).

[0026] During voxel traversal, the ray with the smallest flag is selected first. When the flag bit of a voxel is 0, subsequent ray traversal continues. When the flag bit of a voxel is 1, it indicates that all rays have entered the next quadrant. The starting point and direction of all rays in the next quadrant are reloaded, and the flag bit is set from 1 to 0. For some rays that have finished rendering or penetrated the scene, their flag bit is 2 and does not need to be changed. Then, the next stage of ray traversal continues. When the flag bit of a voxel is 2, it indicates that all rays have finished rendering. The starting point and direction of rays from new rendering blocks can be loaded, and the next stage of ray traversal continues. By introducing flag bits, cross-quadrant and cross-rendering block ray synchronization is achieved.

[0027] Furthermore, the voxel data is reused: the 4×4×4 basic voxels are merged into a macro voxel, and the grid area covered by the macro voxel projected onto each two-dimensional feature plane is merged into a macro grid, so as to reduce the number of times the feature vectors of adjacent voxels are repeatedly loaded at the boundary.

[0028] Furthermore, the skip grid is compressed: a dual-resolution skip grid (e.g., 128 and 32) is used to accelerate ray propagation in empty space. The high-resolution grid is represented as a bitmap and compressed, recording only its distribution within the non-empty regions of the coarse-resolution grid, thus compressing the total memory footprint of the skip grid.

[0029] The present invention also provides a neural radiation field rendering acceleration device, comprising: (1) a block generator: used to initialize rendering blocks, calculate rays and perform initial spatial jumps; (2) a voxel traversal module: responsible for finding the nearest voxel; (3) a ray traversal module: responsible for collecting rays within a specific voxel; (4) a rendering FIFO and sub-block rendering unit: used to buffer and perform ray sampling and feature accumulation; wherein, the sub-block rendering unit is used to perform sampling, interpolation and feature accumulation, and store the results in an intermediate value double buffer; (5) a spatial jump module: responsible for calculating the next voxel of the ray; (6) an MLP engine: used to perform delayed coloring on the accumulated feature vectors to generate the final pixel color; (7) an on-chip memory: used to store master-slave ray information tables, the starting point and direction of rays in different quadrants, jump grid compression bitmaps, rendering intermediate values, macro voxel and macro plane vector information, etc. All modules work together through the master-slave table structure and pipeline control to complete end-to-end real-time rendering.

[0030] Compared with the prior art, the beneficial effects of the present invention are: (1) by storing parameterized ray information (starting point, direction, etc.) instead of complete data of all sampling points, the on-chip storage overhead of each ray is reduced by 80%, which enables the parallel processing of several times the number of rays under the same hardware resources, and greatly saves on-chip storage.

[0031] (2) Based on the ray synchronous propulsion algorithm, the repeated access of macro voxels is effectively reduced; 4×4×4 voxels are merged into a macro voxel and a macro plane mesh of the corresponding projection size is loaded onto the chip, which effectively reduces the repeated access of voxel corner feature vectors and significantly reduces off-chip memory access.

[0032] (3) The master-slave ray information table structure successfully parallelizes the two originally serial processes of voxel traversal and ray space jump, effectively shortening the jump calculation delay, avoiding idle processor resources, and ensuring that the hardware computing unit always operates at high efficiency.

[0033] (4) Through the above algorithm-hardware co-optimization, the present invention can achieve real-time high frame rate neural radiation field rendering in boundless scenes.

[0034] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0035] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be described in detail below with reference to the accompanying drawings, in which: Figure 1 is a pipeline calculation diagram of the present invention; Figure 2 is a data layout diagram of the ray information table; Figure 3 is a chip architecture diagram; Figure 4 is a main and secondary table structure proposed in the present invention; Figure 5 is a rendering flowchart of synchronous execution; Figure 6 is a schematic diagram of quadrant compression introduced by the MERF algorithm; Figure 7 is a graph showing the relationship between rendering latency and the number of sampling points of the accelerator in different scenarios according to the embodiment of the present invention. Detailed Implementation

[0036] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0037] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0038] This invention provides a method and apparatus for accelerating neural radiation field rendering. The invention will be further described in detail below with reference to Figures 1-7.

[0039] A neural radiation field rendering acceleration device mainly includes: (1) a block generator: responsible for initializing image blocks, calculating the starting point and direction of rays in different quadrants within each block, and performing initial spatial jumps to advance the rays to the first non-empty voxel on its path; (2) a voxel traversal module: responsible for traversing the ray information table and finding the voxel_id closest to the camera among all rays; (3) a ray traversal module: receiving the target voxel_id output by the voxel traversal module and searching for all ray segments located within that voxel from the ray information table; (4) a rendering FIFO. : A first-in-first-out buffer is used to temporarily store the ray segments to be rendered found by the ray traversal module; (5) Sub-block rendering unit: Takes out the ray segments from the rendering FIFO, samples them according to their step count, obtains voxel features and color features, and accumulates feature vectors, color and volume density values; (6) Spatial jump module: Receives the ray segments that have completed the rendering of the current voxel, calculates the next non-empty voxel through the ray-bounding box intersection test, and updates the ray information; (7) MLP engine: A dedicated processing unit, usually containing a PE array, is used to perform deferred shading. It reads the accumulated feature vectors and colors from the intermediate value double buffer of the sub-block rendering unit, and calculates the final compensation color in combination with the ray direction; (8) On-chip memory: Used to store master-slave ray information table, the starting point and direction of the ray in different quadrants, jump mesh compression bitmap, rendering intermediate values, macro voxel and macro plane vector information, etc.

[0040] All modules are controlled and work together through master-slave ray information tables and pipelines to complete end-to-end real-time rendering.

[0041] A method for accelerating neural radiation field rendering: Figure 1 shows the pipeline computation diagram of this method. First, the block is initialized, and rays and spatial jumps are calculated to the first non-empty voxel. Then, the block is rendered, and the ray information for the next block is pre-calculated. After the entire block is rendered, the rays in the block are sequentially input into the MLP for deferred shading. During the block rendering stage, voxel traversal and ray traversal are performed sequentially. After the two traversal processes are completed, the next round of traversal immediately begins, while the ray segments of the previous round are rendered and spatial jumps are performed.

[0042] Specifically, this includes: Step 1: Parameterized storage and initialization of ray information. As shown in Figure 2 – the ray information table data layout diagram, the ray is uniformly sampled within voxels. Through AABB bounding box intersection tests, the entry and exit points of the ray and voxel are calculated. The distance between the entry and exit points is divided by a fixed single step length to obtain the number of steps within the voxel. When a sampling point in the next voxel is needed, only the ray travel distance, voxel_id, and number of steps need to be updated. Therefore, for a sampling point within a single voxel of a ray, given its entry point, direction, and number of steps, all sampling points of the ray within the voxel can be directly calculated.

[0043] Therefore, on-chip storage of sampling points is no longer necessary; only a ray information table needs to be maintained. Each entry includes the pixel_id corresponding to the ray, origin, direction, ray travel distance, voxel_id, and the number of steps within the voxel (times). Compared to storing an average of 65 sampling points per ray, this ray-based storage method significantly reduces on-chip storage consumption.

[0044] Figure 3 shows the chip architecture. During initialization, the block generator first calculates all the rays generated by a block of the image in different quadrants, performs spatial jumps on the rays to reach the first voxel, and updates the ray information table to record the current position of the ray.

[0045] Step 2: Ray Synchronous Advancement and Master-Slave Table Collaborative Workflow To adapt the rendering process to on-chip storage, this invention proposes a ray synchronous advancement workflow. This involves traversing the ray information table, finding the voxel closest to the camera, rendering all ray segments within that voxel, and then synchronously jumping to the next voxel to avoid duplicate voxel loading.

[0046] While the aforementioned synchronous ray propagation method can reduce on-chip storage space and increase voxel sharing among rays, thereby reducing off-chip voxel reloading, it requires waiting for all traversed rays to complete their spatial jumps before the next voxel traversal can begin. This significantly increases on-chip traversal computation latency. To prevent voxel traversal from waiting for spatial jumps to complete before starting the next voxel traversal, this invention proposes a master-slave ray information table structure working collaboratively. The master table records the voxel where the ray is currently located, and the slave table records the next voxel the ray is about to enter, enabling parallel processing of voxel traversal and spatial jumps, reducing computational latency. Figure 4 shows the master-slave table structure, with the blue table being the master table and the yellow table being the slave table. The master table records the voxel where the ray is currently located, and the slave table records the next voxel the ray will reach.

[0047] a. Voxel Traversal: After the spatial jump is completed, voxel traversal begins. As shown in Figure 5 - Synchronous Rendering Flow, the voxel traversal module starts scanning the main ray information table and traversing in parallel to compare the voxel identifiers id in the ray information table. First, the flag bits are compared, with the smallest flag bit taking priority. If the flag bits are the same, the voxel_id closest to the camera's starting point is found and off-chip prefetching is performed.

[0048] Using the nearest voxel_id as the depth-first principle, the depth coordinate axis is determined by the longest component in the direction of the camera's starting point. The relative distance from the target voxel to the voxel where the camera is located is calculated as: |x-x0|*128*128+|y-y0|*128+|z-z0|, where the x-axis is the depth direction, which can be flexibly changed according to the actual depth coordinate axis; x, y, and z are the voxel coordinates of the current ray, and x0, y0, and z0 are the voxel coordinates of the camera.

[0049] b. Ray Collection and Submission: The voxel traversal module sends the most recently found voxel_id to the ray traversal module, instructing it to search for all ray segments of the corresponding voxel. The ray traversal module then traverses the main table, inputting the ray information entry corresponding to the voxel_id into the rendering FIFO. It then fills the empty positions in the main table with the corresponding ray entries from the slave table to maintain the compactness of the main table and prepare for the next round of voxel traversal. Simultaneously, it sets the slave table ray entry position to a pending state (i.e., setting the voxel_id flag to 3), awaiting space jump updates. After ray traversal is complete, the main table is updated, allowing the next round of voxel traversal to proceed. During ray traversal of the main table, if the ray at the slave table position corresponding to the target voxel_id is in a pending state, it enters a spin state to wait for the slave table ray information to be updated.

[0050] c. Parallel rendering and skipping: The sub-block rendering unit actively retrieves ray segments from the FIFO, performs step sampling according to times, performs trilinear interpolation and bilinear interpolation on the sampled points, accumulates the feature vector, color, and volume density, and then performs volume rendering and puts them into the intermediate value double buffer.

[0051] The same ray segment is also input into the spatial jump module, which calculates the next voxel the ray will enter after exiting the current voxel, based on the jump grid the ray is currently in. The new distance, voxel_id, times, etc., are written back to the corresponding positions in the ray information table. With the update of voxel_id, the pending state of the ray is naturally released.

[0052] d. Looping and Termination: In steps b and c, the voxel traversal module retraces the ray information table based on the updated master table to find all rays in the voxel and send them to the backend rendering. At the same time, it moves these rays to the next voxel and updates the distance, voxel_id, and times in the ray information table.

[0053] Repeat the above process until the voxel traversal module finds that the voxel_id flag of all surviving rays is 2, which indicates that the current image block has been rendered.

[0054] After the entire block is rendered, the Multilayer Perceptron (MLP) engine starts, retrieves the accumulated feature vector and accumulated color from the double buffer, and inputs the direction vector encoding into the processing unit (PE) array. It then calculates the compensation color and adds it to the accumulated color to obtain the final color. After the entire block is deferred-shaded, it is stored off-chip, awaiting the rendering of the next block.

[0055] In addition, special processing is performed for quadrant compression: As shown in Figure 6 - a schematic diagram of quadrant compression introduced by the MERF algorithm, this invention introduces flag bit processing for quadrant compression in the MERF algorithm to ensure correct synchronization when the ray crosses quadrants.

[0056] First, voxel_id consists of 4 bytes, with the lower three bytes used to store voxel coordinates and the higher byte used as a flag.

[0057] After voxel traversal is complete, rendering is judged based on the voxel_id flag of the selected main table: when the voxel_id flag is 0, it means that the ray is in normal rendering state and ray traversal can be performed.

[0058] When the voxel_id flag is 1, it means that all rays have entered the next quadrant. At this time, the starting point and rays of all rays are reloaded, and the flag of rays with the flag of 1 is set to 0 (some rays have the flag of 2, which do not need to be modified), and the rays of the next quadrant are advanced and rendered. When the voxel_id flag is 2, it means that the entire rendering area has been rendered and the next area can be loaded for rendering.

[0059] During synchronous propagation, when the spatial jump module detects a ray about to enter a new quadrant, it sets its `voxel_id` flag to 1 and resets the `distance`, `times`, and non-flag bits of `voxel_id` to 0. When a ray is detected leaving a quadrant, it sets its `voxel_id` flag to 2 and resets the `distance`, `times`, and non-flag bits of `voxel_id` to 0. During voxel traversal, the system prioritizes processing rays with a flag of 0. Only when all ray flags in the main table are not 0 does the system determine that all rays have reached the boundary of the current quadrant. At this point, the system reloads the start and direction parameters of the new quadrant for all rays with a flag of 1, resets their flags to 0, and then begins synchronous propagation within the new quadrant, ensuring the synchronization of the entire rendering block during quadrant switching.

[0060] During volume rendering, if the ray opacity reaches the threshold, it will also trigger the setting of the voxel_id flag to 2.

[0061] Macro-voxel and macro-mesh data reuse and multi-resolution skip mesh compression optimization: 4 3 Individual voxels are merged into a macro voxel, and the macro voxel is projected to cover 16... 2 Each grid is merged into a macro grid. When loading voxel data, macro voxel / macro grid is used as the unit, which significantly reduces the repeated loading of corner feature vectors of adjacent voxels and reduces random access to voxels.

[0062] To accelerate ray propagation in voxel-free space, the MERF algorithm introduces a multi-resolution skip grid, with a total of five resolutions: [256, 128, 64, 32, 16]. To conserve on-chip storage for the high-resolution skip grid, this embodiment preferably uses a dual-resolution skip grid with coarse and fine 32 and 128 resolutions. The 128-resolution grid is represented as a bitmap and compressed, recording only the non-empty regions of the 32-resolution grid. In this way, the total memory usage of the skip grid is kept to within approximately 50KB.

[0063] Specific implementation results: Figure 7 shows the relationship between the rendering latency and the number of sampling points of this accelerator in different scenarios. The table below shows the rendering frame rate (FPS) of this accelerator in different scenarios simulated on the System C platform.

[0064] Experimental data shows that the accelerator proposed in this invention can achieve a simulation frame rate of over 30fps in multiple scenarios, fully meeting the needs of interactive applications and possessing excellent real-time rendering performance.

[0065]

[0066] In summary, this invention addresses the issues of off-chip memory access and on-chip storage in MERF borderless scene rendering by proposing an algorithm-hardware co-design. It achieves real-time rendering of borderless scenes by replacing sampling point storage with ray information tables, reducing repeated loading of macro voxels using synchronous advancement algorithms, and implementing parallel processing of master-slave tables, combined with macro voxel merging and skip mesh compression.

[0067] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for accelerating neural radiation field rendering, characterized in that, Includes the following steps: Step 1: Storing Parametric Information of Ray Information: A ray information table is maintained on-chip. Each entry in the ray information table stores the parametric information of a single ray. This parametric information includes at least: the voxel_id of the current voxel, the ray origin, the direction, the ray travel distance, and the number of steps (times) within the current voxel; Step 2: Performing Ray Synchronous Rendering: The following steps are iteratively executed on a per-rendering block basis: a. Voxel Traversal: The voxel traversal module traverses the ray information table, finding the voxel_id closest to the camera origin; b. Ray Collection and Submission: The voxel traversal module collects all ray segments located in the voxel_id and inputs the corresponding ray information entries into the rendering FIFO; c. Parallel rendering and spatial jump: The sub-block rendering unit actively retrieves ray segments from the FIFO, performs step sampling, interpolation calculation, feature vector accumulation, and volume rendering, and stores the results in the intermediate value buffer; the ray segments retrieved from the FIFO are simultaneously input into the spatial jump module to calculate their next voxel and update their information in the ray information table; d. Looping and termination: Steps a, b, and c are executed in a loop until the rendering of the current rendering block is completed; after the entire block is rendered, the MLP engine starts and performs deferred shading on the rays in the block.

2. The neural radiation field rendering acceleration method according to claim 1, characterized in that, A master-slave ray information table structure is adopted; wherein, the master ray information table is used to record the voxel information where the ray is currently located, and the slave ray information table is used to record the next voxel information that the ray will enter after a spatial jump; in step b, while collecting the ray segments from the master ray information table and sending them into the rendering FIFO, the ray entries at the corresponding positions in the slave ray information table are immediately filled into the empty positions of the master ray information table.

3. The neural radiation field rendering acceleration method according to claim 2, characterized in that, The entries in the ray information table can be marked as pending. When step b traverses from the main table to the ray segment in the corresponding voxel, if the entry in the secondary table corresponding to the ray segment position is in a pending state, the ray traversal is blocked, ray collection is paused, and the update of the entry is waited for.

4. The neural radiation field rendering acceleration method according to claim 1, characterized in that, The voxel_id contains a flag bit; the flag bit is used to indicate at least one of the following states: normal rendering state, entering a new quadrant state, rendering completed state, and ray pending update state; when a ray enters a new quadrant during a spatial jump, its flag bit is set to the entering new quadrant state; when voxel traversal finds that the flag bits of all rays are in an abnormal rendering state, the starting point, direction, and flag bits of all rays are uniformly reset, and ray traversal is synchronously advanced in the new quadrant or new rendering block.

5. The neural radiation field rendering acceleration method according to claim 1, characterized in that, The method also includes data reuse: merging multiple basic voxels into a macro voxel, and projecting the macro voxel onto the grid area covered by each two-dimensional feature plane to merge it into a macro grid, so as to reduce the number of times the feature vectors of adjacent voxels are repeatedly loaded at the boundary.

6. The neural radiation field rendering acceleration method according to claim 1, characterized in that, The method also includes skip grid compression: a skip grid with coarse and fine resolutions is used, and the high-resolution grid is represented by a bitmap and its distribution in the non-empty areas of the coarse-resolution grid is compressed to reduce memory usage.

7. A neural radiation field rendering acceleration device, characterized in that, To implement the method as described in any one of claims 1-7, the apparatus comprises at least: (1) a block generator for initializing image blocks and rays; (2) a voxel traversal module for traversing the ray information table to find target voxels; (3) a ray traversal module for collecting ray segments within the target voxels; (4) a rendering FIFO and sub-block rendering unit for buffering and performing ray sampling and feature accumulation; (5) a spatial jump module for calculating the information of the ray in the next voxel; (6) an MLP engine for performing delayed coloring on the accumulated feature vectors to generate the final pixel color; and (7) an on-chip memory for storing the master-slave ray information table, the starting point and direction of the ray in different quadrants, the jump grid compression bitmap, rendering intermediate values, macro voxel and macro plane vector information, etc. All modules are controlled by the master-slave ray information table and the pipeline, working collaboratively to complete end-to-end real-time rendering.

8. The neural radiation field rendering acceleration device according to claim 8, characterized in that, The sub-block rendering unit is used to perform sampling, interpolation and feature accumulation, and store the results in an intermediate value double buffer; the MLP engine reads the accumulated feature vector and color from the intermediate value double buffer, and calculates the final compensation color by combining the ray direction.

9. A neural radiation field rendering acceleration device according to claim 8 or 9, characterized in that, The on-chip memory maintains a master-slave ray information table structure; The master-slave ray information table includes: a master ray information table, used for traversal and collection by the voxel traversal module and the ray traversal module; and a slave ray information table, used to receive the next voxel information updated by the space jump module.

Images