Method capable of enhancing data locality of ray packets and sampling points in neural rendering process

Abstract

The technical scheme of the invention discloses a method capable of enhancing data locality of a ray packet and a sampling point in a neural rendering process. The invention provides two key scheduling mechanisms, namely an RP reordering method based on voxel tags and a sampling point out-of-order transmitting method based on an on-chip feature cache hit condition, and designs a matched hardware architecture module. By means of the RP reordering method based on the voxel labels, the number of RP processed by unit voxel label switching can be increased, RP-level data aggregation processing is achieved, repeated loading of the same voxel data is reduced, and the cache hit rate is increased. According to the sampling point out-of-order emission method based on the on-chip feature cache hit condition, out-of-order scheduling can be achieved, EMA delay is hidden, and the throughput rate is increased to 8.45%.

Description

Technical Field

[0001] This invention relates to an optimization method for ray packet (RP) and sampling point scheduling strategies in Neural Radiation Field (NeRF) rendering, and more particularly to a scheduling mechanism and its implementation structure that is hardware-oriented, improves data locality and throughput efficiency, belonging to the fields of computer graphics and artificial intelligence. Background Technology

[0002] Instant Neural Graphics Primitives (Instant-NGP)[1] models have been widely used in high-quality 3D reconstruction and new perspective image synthesis. Its typical rendering process includes five stages: ray generation, localization of sampling points, feature interpolation, neural network inference, and volume rendering. In dedicated hardware accelerators, the mainstream architecture usually manages the rendering process based on ray packets (RP). However, existing designs[2] still have the following two performance bottlenecks: 1. Low on-chip cache hit rate and repeated loading of the same data. Most existing methods follow a concatenated order, processing one RP before moving on to the next. Therefore, different RPs may experience time differences when traversing the same voxel. Simultaneously, during voxel traversal, RPs sequentially access different voxel regions in space according to the path of light propagation. This method leads to frequent voxel switching, causing a decrease in on-chip cache hit rate, repeated loading of voxel data, and increased external memory access (EMA) overhead.

[0003] 2. Pipeline congestion occurs when sampling points access feature data. Current methods mostly process sampling points sequentially, one by one. However, the feature data required by a sampling point is highly uncertain. If the feature data required for a certain sampling point has not yet been read into the on-chip cache, the feature data must first be read from external memory and stored into the on-chip cache. During the aforementioned waiting process for feature loading, processing is paused, reducing throughput.

[0004] References [1] T. Müller, A. Evans, C. Schied, and A. Keller, "Instant neuralgraphics primitives with a multiresolution hash encoding," ACM Trans. Graph., vol. 41, no. 4, pp. 1-15, Jul. 2022, doi: 10.1145 / 3528223.3530127. [2] J. Ryu, H. Kwon, W. Park, Z. Li, B. Kwon, D. Han, D. Im, S. Kim, H. Joo, M. Kim, and HJ Yoo, “NeuGPU: An energy-efficient neural graphics processing unit for instant modeling and real-time rendering on mobile devices,” in IEEE Journal of Solid-State Circuits, vol. 60, no. 1, pp. 99-111, Jan. 2025, doi: 10.1109 / JSSC.2024.3447701. Summary of the Invention The purpose of this invention is to propose a hardware-friendly scheduling strategy that combines RP reordering and out-of-order sampling point delivery, aiming to solve the following key problems: 1. Improve data reuse at the voxel level, reduce the repeated loading of the same voxel data, and reduce the frequency of EMA; 2. Improve the scheduling flexibility of sampling points and reduce pipeline cavitation caused by waiting for feature loading; 3. Provides a clear hardware architecture design path, adapting to the efficient implementation of on-chip scheduling and caching systems.

[0005] To achieve the above objectives, the technical solution of this invention discloses a method to enhance the locality of ray packet and sampling point data during neural rendering, applicable to dedicated hardware accelerators for NeRF-type models. Its key feature is the use of a voxel-label-based ray packet reordering method and a sampling point out-of-order emission method based on on-chip feature cache hits. The voxel-label-based ray packet reordering method includes the following steps: The processing of the ray packet is broken down into multiple stages, each stage corresponding to the voxel label of the voxel where the ray packet is currently located. The switching of X-ray packets occurs after one stage of the X-ray packet processing has ended; According to the voxel labels of the current stage of the ray package, the ray packages are classified and stored in an item structure. Prioritize processing ray packets located within the same voxel; Once the current stage of processing for all ray packets being processed under the current voxel label is complete, switch to the next voxel label; The out-of-order emission method for sampling points based on on-chip feature cache hits includes the following steps: The processing order of sampling points is determined based on the feature hit status of the sampling points. Sampling points that hit the feature cache are processed first, while sampling points that do not hit the cache are temporarily stored and a feature prefetching request is initiated. They are then processed after the data is returned.

[0006] Preferably, in the voxel-label-based ray packet reordering method: Data D of a ray packet PR Upon entering RP-ROU, RP-ROU selects data D in RPSRAM based on the voxel tag RPTag of the voxel where the ray packet is located. PR The storage location of data D for different ray packets with the same voxel tag RPTag. PR They are stored in the same item structure; RP-ROU reads the data D of the ray packet after reordering. CPR Data D CPR Following subsequent processing: If the processing of the ray packet is not yet complete, the corresponding processing operation in the readout process is as follows: the item selector selects the item structure with the fewest ray packets from the item structures corresponding to the currently active voxel labels, and reads the data D of one ray packet from it. CPR The ray packet is then sent to subsequent processing. If the processing of the ray packet has already ended, the readout process corresponds to a retirement operation. First, select the item structure with the fewest placeholders and ray packets in the item structure corresponding to the currently active voxel label, and remove one of the placeholders in that item structure.

[0007] Preferably, the stored data D PR In this process, the item selector first compares the voxel tag RPTag of the input ray packet with the voxel tags of each item structure within it, finding the item structure with the same voxel tag RPTag as the current ray packet. Then, among these item structures, the item selector selects the item structure Emax with the most ray packets, and sets the data D... PR Stored in the address space corresponding to the item structure Emax.

[0008] Preferably, if the currently input ray packet is a brand new, unprocessed ray packet, the storage operation corresponds to the insertion operation; if the currently input ray packet has already been processed, the storage operation corresponds to the reprocessing operation. When both the insertion and reprocessing operations exist, the arbitrator prioritizes the insertion operation.

[0009] Preferably, if the voxel tag RPTag of the current input ray is different from the voxel tags of all items in the item selector, and there is an idle item structure in the item selector, then the idle item structure is selected and the currently input voxel tag RPTag is assigned to that item.

[0010] Preferably, when all processing of the ray included in the item structure corresponding to the currently active voxel tag RPTag is completed, the voxel tag RPTag is switched. Before switching the voxel tag RPTag, whenever the voxel tag RPTag of the ray packet enters RP-ROU, the tag selector compares the current voxel tag RPTag with all voxel tags RTj, j∈[1,N] in the most recent tag recorder: if the current voxel tag RPTag is matched, the voxel tag RTj corresponding to the current voxel tag RPTag is moved to the first position; if the current voxel tag RPTag is not matched, the voxel tags at positions RT1 to RTN-1 in the most recent tag recorder are moved one position to the right, and the current voxel tag RPTag is stored at position RT1. The tag selector reads the most recently stored voxel tags sequentially from the nearest tag recorder in the order of nearest neighbors, and selects the new voxel tag STag by combining the number of ray packets in the item structure corresponding to the voxel tag.

[0011] Preferably, in the out-of-order emission method for sampling points: After a sampling point is generated, before sending it to the feature interpolation unit for feature calculation, the feature interpolation unit pre-reads the control information of the sampling point to determine whether the feature information required by the sampling point is stored in the on-chip feature cache. If the feature required by the sampling point is already stored in the on-chip feature cache, the sampling point is sent to the feature interpolation unit to read the relevant features and calculate the features of the sampling point. If the feature is not found, the sampling point is temporarily stored, and a preload request is initiated. The sampling point is then emitted after the feature data is read from the external memory into the on-chip cache. While waiting to read feature information from off-chip memory, if subsequent sampling points hit the on-chip feature cache, they can be issued first. That is, subsequent sampling points are sent to the feature interpolation unit to read relevant features and calculate the features of the sampling points, thereby realizing the out-of-order advancement of sampling points.

[0012] Preferably, a queue structure is used to manage the sampling point status, sampling point control information and data information base address. The sampling point control information is stored in a register-based feature scheduling item, and the calculation information of the sampling point is stored in an SRAM-based calculation data memory. The base address in the feature scheduling item is the storage address of the calculation information corresponding to the current sampling point in the calculation data memory. When a sampling point hits a feature cache item, the calculation information of the sampling point in the calculation data storage is read out and sent to the sampling point coordinate calculator to generate the sampling position coordinates SamC. The sampling position coordinates SamC are then sent to the feature interpolation unit to calculate the sampling point features.

[0013] To address the challenges of existing real-time neural image primitive models, this invention proposes two key scheduling mechanisms: a voxel-label-based RP reordering method and a sampling point out-of-order emission method based on on-chip feature cache hits. A corresponding hardware architecture module is also designed. The voxel-label-based RP reordering method increases the number of RPs processed per voxel label switch, enabling RP-level data aggregation processing, reducing redundant loading of the same voxel data, and improving cache hit rate. The sampling point out-of-order emission method based on on-chip feature cache hits achieves out-of-order scheduling, hides EMA latency, and improves throughput by 8.45%. Attached Figure Description

[0014] Figure 1 The diagram illustrates the Instant-NGP algorithm. This invention primarily optimizes steps one and two, as shown in the diagram: Indicates the first The most recent voxel label N represents the capacity of the "Recent Tag Recorder"; HS represents the hit status; RNE represents the number of RPs in a single entry; Represents the address of the item; Figure 2 This is a diagram illustrating the voxel traversal process of an RP. An RP needs to traverse multiple voxels, and different RPs will enter the same voxel, as shown in the diagram: This represents the control / calculation data of the RP, which includes sampling point location information; Indicates the coordinates of the sampling point; This represents the control data of the RP output by the OoO-SI module; Index representing a fine voxel; Figure 3 This illustrates the order of data loading and eviction when RP traverses voxels according to the order of light propagation. Figure 4 The figure illustrates the data loading requirements of the RP reordering method proposed in this invention. This indicates the output sequence number of RP after the last traversal; Figure 5 A schematic diagram of the hardware architecture of RP-ROU Figure 6 A schematic diagram of the hardware architecture for a voxel tag selector; Figure 7 This diagram illustrates two sampling point emission methods. Figure 8 This is a schematic diagram of the OoO-SI architecture, showing the connection between OoO-SI and the feature interpolation unit, and how the emission timing of the sampling point is affected by the feedback signal of the feature interpolation unit. Figure 9This is a schematic diagram of the hardware architecture of the location mover. The item selection signal indicates a write / read operation; This represents the RP and the data of the reordered RP. Tags representing RP; Indicates the voxel label after the switch; The voxel label corresponding to the item. Detailed Implementation

[0015] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0016] like Figure 3 As shown, in the unoptimized case, when the RP traverses voxels, data is repeatedly loaded. For example, the data of Vox3, Vox4, and Vox1 needs to be repeatedly loaded and evicted. Another reason why data needs to be evicted is that the on-chip cache has a limited capacity and cannot store all voxel data in the on-chip cache.

[0017] This invention discloses a method to enhance the locality of ray packets and sampling point data during neural rendering, proposing the following two key scheduling mechanisms and designing a corresponding hardware architecture module. The first key scheduling mechanism is the RP reordering method based on voxel tags, such as... Figure 4 As shown.

[0018] First, the processing of a RP is broken down into multiple stages, each corresponding to a voxel tag of the RP's current location. For example, if an RP traverses Nv voxels, then the processing of that RP will consist of Nv stages.

[0019] Secondly, the switching of RPs occurs after one phase of RP processing has ended, rather than after the entire RP processing has ended. This allows multiple RPs to potentially reside in the same voxel.

[0020] Furthermore, this method categorizes RP data according to the voxel labels at the current stage. RP data under different voxel labels are stored using an "entry" structure.

[0021] Finally, this method prioritizes processing RPs located in the same voxel.

[0022] Once the current stage of processing for all RPs under this voxel label has finished, switch to the next voxel label.

[0023] Specifically, such as Figure 5 As shown, when an RP's data D PR After entering the RP-ROU, the RP-ROU needs to select the storage location, i.e., the address, for the data of the RP in "RPSRAM". The address selection is based on the RP's voxel tag, RPTag. RPs with the same voxel tag are stored in the same "item".

[0024] The "item selector" first compares the voxel label RPTag of the input RP with the voxel label of each item within it, finding items that have the same voxel label as the RP's voxel label RPTag. Then, among these items, it selects the item Emax, which has the most RPs, and stores the RP's data D. PR Store it in the address space corresponding to the item Emax.

[0025] If the current input RP is a brand new, unprocessed RP, the above operation corresponds to the "insert" operation. If the current input RP has already been processed, the above operation corresponds to the "reprocess" operation.

[0026] When both "insert" and "reprocess" operations exist, the arbitrator takes priority over the "insert" operation.

[0027] If the voxel tag RPTag of the currently input RP is different from the voxel tags of all items in the "Item Selector", and there are empty items in the "Item Selector", then select the empty item and assign the currently input voxel tag RPTag to that item.

[0028] In addition to the RP data D PR When writing to "RPSRAM", RP-ROU also needs to read the reordered RP data D. CPR Data D CPR This will undergo further processing. If the processing of RPs is not yet complete, the reading process corresponds to the "processing" operation. The "item selector" selects the item with the fewest RPs from the items corresponding to the currently active voxel tags, and reads the data D of one RP from it. CPR The RP is then sent for further processing. If the RP processing has already finished, the read process corresponds to a "retirement" operation. In this case, the item with the fewest placeholders and RPs among the currently active voxel tags is selected, and one of the placeholders in that item is removed.

[0029] The need for placeholders within items is due to the phased processing method. Without placeholders, when an RP (RPA, assuming its corresponding voxel label is VoxA) is read out by the "processing" operation for subsequent processing, too many RPs might be stored in the items corresponding to VoxA through "insertion" or "reprocessing" operations. In extreme cases (when all items are full), the RPA will be unable to proceed to the next step of processing through the "reprocessing" operation. Another scenario is when an RP completes the traversal of a voxel and enters the next voxel, which corresponds to two operations: "insertion" and "retirement". At this time, since the voxel label RPTag of the RP has changed (e.g., from VoxB to VoxC), a new storage address needs to be allocated to R in "RPSRAM", so the "insertion" operation (for VoxC) is necessary. At the same time, it is necessary to clear a placeholder in the item corresponding to the previous label (VoxB), i.e., the "retirement" operation is necessary.

[0030] The "item selector" supports non-uniform load, where a voxel can be assigned multiple items based on the number of RPs currently in the processing state, preventing resource waste or scheduling blockage.

[0031] like Figure 6 As shown, when the processing of all RPs in the item corresponding to the currently active voxel tag RPTag is finished, the tag needs to be switched. Before this, whenever the voxel tag RPTag of an RP enters RP-ROU, the "tag selector" compares the current voxel tag RPTag (also represented as ETag) with all voxel tags RTj, j∈[1,N] in the "recent tag recorder". If ETag matches, the voxel tag RTj corresponding to ETag is moved to the first position (RT1). If ETag does not match, the voxel tags from the original positions RT1 to RTN-1 are all moved one position to the right, and ETag is stored at the position of RT1. The "tag selector" reads the most recently stored tags from the "recent tag recorder" in the order of nearest neighbors, and selects the voxel tag STag after switching, based on the number of RPs in the item corresponding to the tag.

[0032] The second key scheduling mechanism is the Out-of-Order Sample Issuance (OoO-SI) method based on on-chip feature cache hits.

[0033] The OoO-SI method determines the processing order of sampling points based on their feature hit status. It prioritizes processing sampling points that have hit the feature cache, while sampling points that have not hit the cache are temporarily stored, and a feature prefetch request is initiated. They are then processed after the data is returned. This decouples the sampling point scheduling from the feature loading process, effectively improving feature reuse rate, reducing pipeline cavitation caused by waiting for feature loading, and alleviating congestion.

[0034] like Figure 7 As shown, existing methods for sequentially processing sampling points require waiting for the features needed for a sampling point to be read back from external memory when those features are not in the on-chip cache, thus reducing the processing speed of sampling points. The OoO-SI method proposed in this invention can effectively alleviate the above problems.

[0035] like Figure 8 As shown, after a sampling point is generated, before sending it to the next module (feature interpolation unit) for feature calculation, the feature interpolation unit pre-reads the control information DRP_sP_ctrl of the sampling point to determine whether the feature information required by the sampling point is stored in the on-chip feature cache. If the feature required by the sampling point is already stored in the on-chip feature cache (i.e., the on-chip feature cache is hit), the sampling point is sent to the feature interpolation unit to read the relevant features and calculate the features of the sampling point. If the feature is not hit, the sampling point is temporarily stored, and a "preload" request is initiated. The sampling point is then launched after the feature data is read from external memory (e.g., DDR) into the on-chip cache. At the same time (i.e., while waiting to read the feature information from external memory), if subsequent sampling points hit the on-chip feature cache, they can be launched first. That is, subsequent sampling points are sent to the feature interpolation unit to read the relevant features and calculate the features of the sampling points, thereby realizing the out-of-order advancement of sampling points.

[0036] like Figure 9 As shown, this invention employs a queue structure (position mover) to manage the sampling point status (whether the feature cache is hit), sampling point control information (DRP sz_ctrl), and data information base address. Since the sampling point's computational information (DRP SP_comp) requires more storage space than its control information (DRR_sP_ctrl), we store the control information (DRP sP_ctrl) in a register-based feature scheduling entry, while storing the computational information (DRP_S_comp) in an SRAM-based "computational data memory." The "base address" in the feature scheduling entry is the storage address of the computational information corresponding to that sampling point in the "computational data memory." When a sampling point hits the feature cache entry, the computational information (DRr.sz_comp) in the "computational data memory" is read out and then sent to the "sampling point coordinate calculator" to generate the sampling position coordinates (SamC). The sampling position coordinates (SamC) are then sent to the feature interpolation unit to calculate the sampling point features.

[0037] The queue structure of the position mover supports dynamic scheduling priority adjustment, ensuring that sampling points with feature hits have higher priority. This mechanism essentially decouples feature prefetching from feature hit scheduling, not only reducing EMAs but also effectively hiding the EMA latency caused by cache misses.

[0038] The above implementation methods have been verified in the EDR-NR chip prototype, and have complete hardware mapping and engineering feasibility, and are applicable to inference acceleration of various NeRF series models.

Claims

1. A method for enhancing the locality of ray packets and sampling point data during neural rendering, applicable to dedicated hardware accelerator implementations of NeRF-type models, characterized in that, A method for reordering ray packets based on voxel tags and a method for out-of-order emission of sampling points based on on-chip feature cache hits are adopted. The method for reordering ray packets based on voxel tags includes the following steps: the processing of ray packets is refined into multiple stages, each stage corresponding to the voxel tag of the voxel where the ray packet is currently located; the switching of ray packets occurs after the processing of a stage of the ray packet is completed; the ray packets are classified according to the voxel tag of the current stage of the ray packet, and the ray packet data under different voxel tags are stored in an item structure; ray packets in the same voxel are processed first; after the processing of the current stage of all ray packets under the current voxel tag is completed, the process switches to the next voxel tag. The method for out-of-order emission of sampling points based on on-chip feature cache hits includes the following steps: the processing order of sampling points is determined based on the feature hit status of the sampling points, sampling points that hit the feature cache are processed first, and sampling points that do not hit the cache are temporarily stored and a feature prefetching request is initiated, and they are processed after the data is returned.

2. The method for enhancing the locality of ray packets and sampling point data during neural rendering as described in claim 1, characterized in that, In the voxel-label-based ray packet reordering method: the data D of a ray packet PR Upon entering RP-ROU, RP-ROU selects data D in RPSRAM based on the voxel tag RPTag of the voxel where the ray packet is located. PR The storage location of data D for different ray packets with the same voxel tag RPTag. PR They are stored in the same item structure; RP-ROU reads the data D of the reordered ray packet. CPR Data D CPR Following subsequent processing: If the processing of the ray packet is not yet complete, the corresponding processing operation in the readout process is as follows: the item selector selects the item structure with the fewest ray packets from the item structures corresponding to the currently active voxel labels, and reads the data D of one ray packet from it. CPR The ray packet is then sent to subsequent processing. If the processing of the ray packet has already ended, the readout process corresponds to a retirement operation. First, select the item structure with the fewest placeholders and ray packets in the item structure corresponding to the currently active voxel label, and remove one of the placeholders in that item structure.

3. The method for enhancing the locality of ray packets and sampling point data during neural rendering as described in claim 2, characterized in that, Stored data D PR In this process, the item selector first compares the voxel tag RPTag of the input ray packet with the voxel tags of each item structure within it, finding the item structure with the same voxel tag RPTag as the current ray packet. Then, among these item structures, the item selector selects the item structure Emax with the most ray packets, and sets the data D... PR Stored in the address space corresponding to the item structure Emax.

4. A method for enhancing the locality of ray packets and sampling point data during neural rendering as described in claim 3, characterized in that, If the currently input ray packet is a brand new, unprocessed ray packet, the store operation corresponds to the insert operation. If the currently input ray packet has already been processed, the store operation corresponds to the reprocess operation. When both insert and reprocess operations exist, the arbitrator takes priority over the insert operation.

5. The method for enhancing the locality of ray packets and sampling point data during neural rendering as described in claim 3, characterized in that, If the voxel tag RPTag of the current input ray is different from the voxel tags of all items in the item selector, and there is an empty item structure in the item selector, then the empty item structure is selected and the voxel tag RPTag of the current input is assigned to that item.

6. A method for enhancing the locality of ray packets and sampling point data during neural rendering as described in claim 3, characterized in that, When all processing of the ray included in the item structure corresponding to the currently active voxel tag RPTag is completed, the voxel tag RPTag is switched. Before switching the voxel tag RPTag, whenever the voxel tag RPTag of the ray packet enters RP-ROU, the tag selector compares the current voxel tag RPTag with all voxel tags RTj in the most recent tag recorder, j∈[1,N]: if the current voxel tag RPTag is matched, the voxel tag RTj corresponding to the current voxel tag RPTag is moved to the first position. If the current voxel tag RPTag is not matched, the voxel tags at positions RT1 to RTN-1 in the nearest tag recorder are shifted one position to the right, and the current voxel tag RPTag is stored at position RT1. The tag selector reads the most recently stored voxel tags from the nearest tag recorder in the order of nearest neighbors, and selects the switched voxel tag STag by combining the number of ray packets in the item structure corresponding to the voxel tag.

7. The method for enhancing the locality of ray packets and sampling point data during neural rendering as described in claim 1, characterized in that, In the out-of-order emission method of the sampling points: after the sampling points are generated, before sending the sampling points to the feature interpolation unit for feature calculation, the feature interpolation unit pre-reads the control information of the sampling points to determine whether the feature information required by the sampling points is stored in the on-chip feature cache; if the features required by the sampling points are already stored in the on-chip feature cache, the sampling points are sent to the feature interpolation unit to read the relevant features and calculate the features of the sampling points. If a sample point is not hit, it is temporarily stored and a preload request is initiated. The sample point is then issued after the feature data is read from the external memory into the on-chip cache. While waiting to read the feature information from the external memory, if a subsequent sample point hits the on-chip feature cache, it can be issued first. That is, the subsequent sample points are sent to the feature interpolation unit to read the relevant features and calculate the features of the sample points, thereby realizing the out-of-order advancement of the sample points.

8. A method for enhancing the locality of ray packets and sampling point data during neural rendering as described in claim 7, characterized in that, A queue structure is used to manage the sampling point status, sampling point control information, and data information base address. The sampling point control information is stored in a register-based feature scheduling item, and the calculation information of the sampling point is stored in an SRAM-based computational data memory. The base address in the feature scheduling item is the storage address of the calculation information corresponding to the current sampling point in the computational data memory. When a sampling point hits a feature cache item, the calculation information of the sampling point in the computational data memory is read out and sent to the sampling point coordinate calculator to generate the sampling position coordinates SamC. The sampling position coordinates SamC are sent to the feature interpolation unit for calculating the sampling point features.

Images