A data prefetching method and device for NPU post-processing

Abstract

This invention relates to the field of neural network processor technology, and in particular to a data prefetching method and apparatus for NPU post-processing. By monitoring the amount of unconsumed data in the prefetch memory space in real time, and using the comparison between this data amount and the memory space capacity as the sole basis for triggering subsequent data movement, data movement no longer depends on the current execution progress of the computation pipeline, achieving complete decoupling between data loading and vector computation. In this way, external memory access can be initiated in advance before the computation consumes the current data, fully overlapping with the computation process in time, effectively masking memory access latency, avoiding idle cycles in the computation pipeline due to waiting for data, significantly improving the computational resource utilization and overall throughput performance of NPU post-processing. The entire prefetching control logic is completed automatically by hardware, without software intervention.

Description

Technical Field

[0001] This invention relates to the field of neural network processor technology, and in particular to a data prefetching method and apparatus for NPU post-processing. Background Technology

[0002] After completing major computational tasks such as convolution and matrix multiplication, neural network processing units (NPUs) typically need to perform a series of post-processing operations, such as element-wise vector computations including element-wise addition and subtraction, multiplication, activation functions, and pruning. These computations are usually performed by a dedicated post-processing vector computation pipeline within the NPU. While this pipeline has low computational density, it is highly sensitive to the continuity of data supply, and memory access latency often becomes a performance bottleneck.

[0003] In existing technologies, element-wise vector computation in NPU post-processing generally adopts a "computation-driven data transfer" implementation. Specifically, the NPU post-processing pipeline reads tensor data to be processed from external memory as needed based on the current computation progress. When the computation pipeline finishes consuming the current batch of data, the execution control logic synchronously triggers the Direct Memory Access (DMA) module to transfer the next batch of tensor data from external memory to the on-chip buffer, and then the computation pipeline performs element-wise operations on that batch of data.

[0004] However, the aforementioned existing technologies have the following drawbacks. First, since external data movement is directly triggered by the execution progress of the elementwise computation pipeline, DMA cannot initiate external storage access before the computation consumes data. When the external storage access latency is high or bandwidth is limited, the computation pipeline inevitably idles while waiting for new data, leading to a decrease in computational resource utilization. Second, the computational complexity of a single data element in elementwise vector computation is low, and the pipeline is extremely sensitive to the continuity of data supply. Existing computation-driven movement methods cannot achieve sufficient time overlap between external storage access and internal computation, thus directly exposing external memory access latency to the computation path. Third, the strong coupling between data movement and computation progress limits the scheduling flexibility of DMA, making it difficult to dynamically adjust the movement rhythm according to the real-time status of the buffer, especially when the tensor size is large or the shape is irregular.

[0005] Therefore, how to achieve efficient overlap between external data transfer and internal computation in NPU post-processing elementwise vector computation has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] In view of this, the purpose of the present invention is to provide a data prefetching method and apparatus for NPU post-processing.

[0007] In a first aspect, embodiments of the present invention provide a data prefetching method for NPU post-processing, the method comprising:

[0008] In the NPU's on-chip general-purpose buffer, a prefetch memory space of a predetermined size is allocated for elementwise vector computation;

[0009] Monitor the amount of data in the prefetch memory that has been prefetched but not yet consumed by the elementwise vector computation pipeline. The amount of data increases after each successful fetch of a set of data and decreases after each complete consumption of a set of data.

[0010] The data transfer module moves a set of elementwise computation data from external memory to prefetch memory space, and updates the data volume to increase it after the transfer is completed.

[0011] The elementwise vector computation pipeline reads and consumes a set of data from the prefetch memory, performs element-wise vector computation, and updates the data volume to reduce it after consumption is complete.

[0012] If the amount of monitored data is less than the capacity of the prefetch storage space, it is determined that there is free space, and the next set of elementwise calculation data is moved from the external storage to the prefetch storage space again through the data transfer module, until all elementwise data to be calculated is moved and consumed.

[0013] In conjunction with the first aspect, the step of monitoring the amount of data in the prefetch memory that has been prefetched but not yet consumed by the elementwise vector computation pipeline includes:

[0014] The number of data sets that have been prefetched but not yet fully consumed in the prefetch storage space is recorded in real time using a hardware counter.

[0015] After each successful transfer of a set of data, the hardware counter is incremented by one to update the data volume;

[0016] After each complete consumption of a set of data, the count value of the hardware counter is decremented by one to update the data volume.

[0017] In conjunction with the first aspect, the steps of moving a set of elementwise computation data from external memory to prefetch memory using a data transfer module include:

[0018] The tensor to be computed using elementwise methods is divided into multiple data blocks along at least one spatial dimension;

[0019] The data is sequentially moved from the external memory to the prefetch memory via the data transfer module. Each data block constitutes a set of elementwise computation data.

[0020] In conjunction with the first aspect, it also includes:

[0021] For boundary data blocks located at tensor boundaries and whose data size is less than a complete data block, the data transport module only transports the valid data in the boundary data block and marks the valid data length of the boundary data block in the prefetch storage space.

[0022] The elementwise vector computation pipeline reads and consumes data from the prefetch memory based on the effective data length of the boundary data block.

[0023] In conjunction with the first aspect, the size of the prefetch storage space is dynamically configured based on the type of the currently executed elementwise operator or the size of the tensor to be computed.

[0024] In conjunction with the first aspect, there are multiple data transport modules; these multiple data transport modules work in parallel to support the simultaneous execution of multi-path elementwise vector computation pipelines.

[0025] Each data transfer module corresponds to one elementwise vector computation pipeline, and each pipeline is independently equipped with prefetch storage space and a hardware counter for monitoring data volume.

[0026] In conjunction with the first aspect, if the monitored data volume is equal to the capacity of the prefetch storage space, it is determined that the prefetch storage space is full, and the data transfer module suspends transfer until the elementwise vector calculation pipeline consumes data so that the data volume is less than the capacity, at which point transfer resumes.

[0027] In conjunction with the first aspect, the size of the data block is matched to the width of the data group processed in a single operation of the elementwise vector computation pipeline.

[0028] In conjunction with the first aspect, the step of monitoring the amount of data in the prefetch memory that has been prefetched but not yet consumed by the elementwise vector computation pipeline includes:

[0029] Calculate the difference between the write pointer and the read pointer; where the write pointer points to the write position of the latest set of data in the prefetch memory, and the read pointer points to the read position of the latest set of data that was consumed.

[0030] The difference is used as the amount of data in the prefetch memory that has been prefetched but not yet consumed by the elementwise vector computation pipeline.

[0031] Secondly, the present invention also provides a data prefetching apparatus for NPU post-processing, the apparatus being used to perform the method described above; the apparatus includes:

[0032] An on-chip general-purpose buffer is provided, which includes a prefetch memory space of a predetermined size. The prefetch memory space is used to store elementwise computation data prefetched from external memory and is connected to the data read port of the elementwise vector computation pipeline.

[0033] The data volume monitoring unit is used to monitor the amount of data that has been prefetched in the prefetch storage space but has not yet been consumed by the elementwise vector computation pipeline. The data volume increases after each successful transfer of a set of data and decreases after each complete consumption of a set of data.

[0034] The data transfer module is connected to the external memory and the prefetch memory space. It is used to transfer a set of elementwise computation data from the external memory to the prefetch memory space, and after the transfer is completed, it triggers the data volume monitoring unit to update the data volume to increase it.

[0035] The elementwise vector computation pipeline is used to completely read and consume a set of data from the prefetch memory, perform element-wise vector computation, and trigger the data volume monitoring unit to update the data volume to reduce it after consumption is completed.

[0036] The control unit, connected to the data volume monitoring unit and the data transfer module, is used to determine that there is free space when the monitored data volume is less than the capacity of the prefetch storage space, and to control the data transfer module to transfer the next set of elementwise calculation data from the external memory to the prefetch storage space again, until all the elementwise data to be calculated has been transferred and consumed.

[0037] The embodiments of the present invention bring the following beneficial effects: The present invention provides a data prefetching method and apparatus for NPU post-processing. The method includes: allocating a prefetch storage space of a predetermined size in the on-chip general-purpose buffer of the NPU for elementwise vector computation; monitoring the amount of data prefetched in the prefetch storage space but not yet consumed by the elementwise vector computation pipeline, the amount of data increasing after each successful transfer of a set of data and decreasing after each complete consumption of a set of data; transferring a set of elementwise computation data from external memory to the prefetch storage space through a data transfer module, and updating the amount of data to increase it after the transfer is completed; the elementwise vector computation pipeline completely reads and consumes a set of data from the prefetch storage space, performs element-wise vector computation, and updates the amount of data to decrease it after the consumption is completed; if the monitored amount of data is less than the capacity of the prefetch storage space, it is determined that there is free space, and the next set of elementwise computation data is transferred from external memory to the prefetch storage space again through the data transfer module, until all elementwise data to be computed is transferred and consumed.

[0038] This invention monitors the amount of unconsumed data in the prefetch storage space in real time and uses the comparison between this data amount and the storage space capacity as the sole basis for triggering subsequent data movement. This eliminates the dependence of data movement on the current execution progress of the computation pipeline, achieving complete decoupling between data loading and vector computation. In this way, external memory access can be initiated in advance before the computation consumes the current data, fully overlapping with the computation process in time. This effectively masks memory access latency, avoids idle time in the computation pipeline due to waiting for data, improves the utilization of NPU post-processing computational resources and overall throughput performance, and the entire prefetch control logic is completed automatically by hardware without software intervention.

[0039] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0040] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0041] Figure 1 This is a flowchart illustrating a data prefetching method for NPU post-processing.

[0042] Figure 2 This is a schematic diagram of a data prefetching device for NPU post-processing.

[0043] Figure label:

[0044] 10 - On-chip general-purpose buffer, 20 - Data volume monitoring unit, 30 - Data transfer module, 40 - Elementwise vector computation pipeline, 50 - Control unit. Detailed Implementation

[0045] To facilitate understanding of this embodiment, the technical terms used in this invention will be briefly introduced below.

[0046] NPU (Neural Processing Unit): A processor specifically designed to accelerate neural network computations, typically containing main computation units such as convolution and matrix multiplication, as well as post-processing vector computation units.

[0047] Element-wise vector computation refers to operations that perform the same operation independently on each element of a tensor, such as element-wise addition, multiplication, ReLU activation function, numerical pruning, etc. In an NPU, these operations are typically executed in parallel using a post-processing vector computation pipeline in a SIMD manner.

[0048] Prefetch memory: A dedicated area within the NPU's on-chip general-purpose buffer is used to temporarily store elementwise computation data pre-moved from external memory, enabling the computation pipeline to read data with low latency.

[0049] Data transfer module: A hardware unit responsible for transferring data between external memory and on-chip buffers, typically implemented as a direct memory access (DMA) controller.

[0050] Unconsumed data volume: refers to the number of data sets in the prefetch storage space that have been prefetched but have not yet been fully consumed by the computation pipeline, used to characterize the current occupancy depth of the buffer.

[0051] Decoupled prefetcher / control unit: Hardware control logic that triggers data movement independently of computation progress, based on the relationship between the amount of unconsumed data in the buffer and its capacity.

[0052] After introducing the technical terms involved in this invention, the application scenarios and design concepts of the embodiments of this invention will be briefly described below.

[0053] In existing NPU post-processing, data transfer for elementwise vector computation is triggered synchronously with the computation progress, directly exposing external memory access delays. The computation pipeline is prone to idle while waiting for data, resulting in low resource utilization.

[0054] Based on this, embodiments of the present invention provide a data prefetching method and apparatus for NPU post-processing.

[0055] Example 1

[0056] This invention provides a data prefetching method for NPU post-processing, combined with Figure 1 As shown, the method includes:

[0057] S110 allocates a prefetch memory space of a predetermined size in the NPU's on-chip general-purpose buffer for elementwise vector computation.

[0058] S120 monitors the amount of data in the prefetch storage space that has been prefetched but not yet consumed by the elementwise vector computation pipeline. The amount of data increases after each successful transfer of a set of data and decreases after each complete consumption of a set of data.

[0059] S130: The data transfer module moves a set of elementwise computation data from the external memory to the prefetch storage space, and updates the data volume to increase it after the transfer is completed.

[0060] S140, the elementwise vector computation pipeline reads and consumes a set of data from the prefetch memory, performs element-wise vector computation, and updates the data volume to reduce it after consumption.

[0061] S150, if the monitored data volume is less than the capacity of the prefetch storage space, it is determined that there is free space, and the next set of elementwise calculation data is moved from the external memory to the prefetch storage space again through the data transfer module, until all elementwise data to be calculated is moved and consumed.

[0062] This invention allocates prefetch memory space within the NPU's on-chip general-purpose buffer and monitors the amount of prefetched but unconsumed data in this space in real time. This allows the data transport module to update the data volume after each data transport, increasing the volume, while the computation pipeline updates the data volume after each data transport, decreasing it. Furthermore, once the current data volume is detected to be less than the prefetch memory space's capacity, indicating the existence of free space, the next data transport is triggered independently of the current execution progress of the computation pipeline, and this process is repeated until all data is processed. Thus, data transport no longer passively waits for computation but is initiated in advance based on the buffer's idle state, allowing external memory access and internal computation to fully overlap in time. This effectively masks memory access latency, avoids idle cycles in the computation pipeline due to data interruptions, and significantly improves the utilization of NPU post-processing computational resources and overall throughput performance.

[0063] The NPU's on-chip general-purpose buffer is typically used to temporarily store intermediate data during computation. Step S110 allocates a dedicated area as a prefetch storage space, the size of which can be pre-set according to the computational requirements of the current elementwise operator or the size of the tensors to be processed. For example, it can be set to accommodate 4, 8, or more sets of vector data.

[0064] Once allocated, this prefetch memory space is connected to the data read port of the subsequent elementwise vector computation pipeline, allowing the pipeline to read data directly from this space without accessing external memory each time. By pre-allocating a fixed-size buffer region, a clear hardware resource foundation is provided for subsequent data prefetching and flow control.

[0065] In conjunction with the first aspect, step S120 includes:

[0066] S121 uses a hardware counter to record in real time the number of data groups that have been prefetched but not yet fully consumed in the prefetch storage space.

[0067] S122 increments the hardware counter by one after each successful transfer of a set of data to update the data volume.

[0068] S123, after each complete consumption of a set of data, decrements the count value of the hardware counter by one to update the data volume.

[0069] To monitor and accurately track the occupancy status of the prefetch memory in real time, this embodiment employs a hardware counter. This counter is a readable and writable hardware register whose value is always equal to the number of data sets prefetched into the current prefetch memory but not yet fully read and processed by the elementwise vector computation pipeline. For example, if the prefetch memory can hold a maximum of M data sets, the counter's value ranges from 0 to M. Initially, the prefetch memory is empty, and the counter is reset to zero. This hardware counter is directly connected to the control logic of the data transfer module and the computation pipeline, enabling it to respond in real time to transfer completion and consumption completion events, thus dynamically and accurately reflecting the amount of data to be consumed in the buffer. Compared to software-maintained variables, hardware counters offer advantages such as fast response speed, no software overhead, and immunity to CPU interrupt interference, making them particularly suitable for NPU post-processing scenarios with stringent latency and throughput requirements.

[0070] Whenever a data transfer module (such as a DMA controller) completes writing a complete set of elementwise computation data from external memory to the prefetch memory, the hardware immediately generates a transfer completion signal. This signal is sent to the increment input of a hardware counter, triggering the counter to increment by one, changing the count value from the current value to the current value +1. This update operation is strictly synchronized with the transfer completion event, ensuring that the counter value is refreshed at the same moment the data is written to the buffer, thus accurately reflecting that the newly added set of data is available for consumption by the computation pipeline. For example, if the counter value is k before transfer, it means that there are k sets of data waiting to be consumed; after successfully transferring one set, the counter value becomes k+1, indicating that the number of data sets waiting to be consumed has increased by one. This update process requires no software intervention and is completely completed automatically by the hardware logic, ensuring real-time performance and reliability.

[0071] Each time the elementwise vector computation pipeline fully reads a set of data from the prefetch memory and completes the corresponding element-wise vector computation, the hardware generates a completion signal. This signal is sent to the decrement input of the hardware counter, triggering the counter to decrement by one, changing the count value from the current value to -1. This update operation is synchronized with the consumption progress of the computation pipeline, accurately reflecting that a set of data in the buffer has been processed and its occupied storage location has been released to free. For example, if the counter value before consumption is k (k≥1), it means that there are k sets of data to be consumed; after fully consuming a set, the counter value becomes k-1, indicating that the number of data sets to be consumed has decreased by one. Through this decrement operation, the hardware counter can track the change in the free capacity of the buffer in real time, providing accurate input conditions for subsequent judgments on whether to continue prefetching.

[0072] In this way, through the cooperation of S121 to S123, the hardware counter forms a closed-loop feedback mechanism: the transport event increments it, the consumption event decrements it, and the count value is always equal to the number of unconsumed data sets in the buffer.

[0073] In conjunction with the first aspect, step S130 includes:

[0074] S131, divide the tensor to be computed using elementwise into multiple data blocks along at least one spatial dimension.

[0075] S132, the data of each data block is sequentially moved from the external memory to the prefetch memory space through the data transfer module, wherein each data block constitutes a set of elementwise computation data.

[0076] The data transfer module (e.g., a DMA controller) is responsible for transferring the elementwise data to be computed from external memory such as off-chip DDR or HBM to the on-chip prefetch memory. Each transfer operation is based on a "set of data," the size of which can match the data width processed by the computation pipeline in a single operation, or it can use a fixed tile size. Upon successful completion of the transfer operation, the data volume update operation described in S120 is immediately triggered, incrementing the counter representing the amount of unconsumed data by 1. This process is independent of whether the computation pipeline is currently processing data; the transfer can be initiated independently as long as there is free space in the prefetch memory. Timely updates to the data volume after transfer ensure the real-time nature and accuracy of the buffer status information.

[0077] Specifically, for tensors that require post-processing (such as a 3D feature map containing height, width, and number of channels), if the entire tensor is moved from external memory to the on-chip buffer at once, it may cause buffer overflow due to the large size of the tensor, or cause the computation pipeline to wait for a long time due to the long transfer time.

[0078] Therefore, in step S131, the tensor to be computed is divided into multiple data tiles of equal or unequal size along its spatial dimensions (e.g., height, width, or both). Each data tile contains all elements within a contiguous sub-region of the tensor. The size of the data tiles can be pre-set based on factors such as the capacity of the NPU's on-chip prefetch memory, the processing width of the computation pipeline, and the burst transfer characteristics of external memory. For example, a 1024×1024 matrix can be divided into 32 32×1024 data tiles along the row direction, or into multiple 32×32 small squares along both the row and column directions. Through this tile-based strategy, the originally large tensor is decomposed into multiple basic transport units of appropriate granularity, allowing subsequent data transport and computation to be pipelined in a tile-by-tile manner, while also facilitating the handling of misalignment at tensor boundaries.

[0079] After tensor partitioning is completed, in step S131, the data transfer module (e.g., a DMA controller) reads each data block from external memory and writes it into the on-chip prefetch memory space one by one in a predetermined order (e.g., row-major or column-major). Each data block corresponds to a set of elementwise computation data as mentioned in step S110. The transfer module does not need to wait for the computation pipeline to consume the previous data block before starting to transfer the next data block; instead, as long as there is free space in the prefetch memory space, the transfer module can continue to initiate the transfer of the next data block. For example, after the first data block is transferred to the prefetch memory space, if the buffer is not full, the transfer module can immediately transfer the second data block, and so on. The size of the data block can be pre-configured according to factors such as the capacity of the NPU's on-chip prefetch memory space, the processing width of the computation pipeline, and the burst transfer characteristics of the external memory, and is not fixed.

[0080] Simultaneously, the computation pipeline sequentially retrieves data blocks from the prefetch memory for element-wise operations according to its own rhythm. The data transfer and computation processes overlap at the data block level. By sequentially transferring each data block and using a hardware counter in step S120 to track the number of unconsumed data blocks in real time, the system can automatically maintain a certain number of unprocessed data blocks in the buffer, thus preventing the computation pipeline from becoming idle due to waiting for data and also preventing buffer overflow due to excessively fast data transfer. The remaining portions of data blocks at tensor boundaries that are smaller than complete data blocks are also treated as independent data blocks for transfer and processing, and their effective data length is marked for identification by the computation pipeline.

[0081] By using data blocks as the basic operation unit in steps S131-S132, this method can flexibly adapt to tensors of arbitrary size while maximizing the use of on-chip buffer resources and external storage bandwidth.

[0082] In conjunction with the first aspect, the method also includes:

[0083] S133, for boundary data blocks located at tensor boundaries and whose data size is less than a complete data block, the data transport module only transports the valid data in the boundary data block and marks the valid data length of the boundary data block in the prefetch storage space.

[0084] S134, the elementwise vector computation pipeline reads and consumes the boundary data block from the prefetch memory space based on the effective data length.

[0085] When a tensor is divided into multiple data blocks according to S131, the total size of the tensor is not necessarily an integer multiple of the data block size. Therefore, a boundary data block, smaller than the complete data block, will remain at the end of the tensor. For example, in a matrix with a width of 100, if the data block width is 32, the first three data blocks will cover columns 1-32, 33-64, and 65-96 respectively, leaving columns 97-100 (only 4 columns) as a boundary data block. If such boundary data blocks are still processed according to the size of the complete data block, data exceeding the actual range of the tensor will be read from external memory, leading to data errors or invalid access.

[0086] To avoid this problem, the data transfer module first identifies the valid data range of the boundary data block (e.g., calculating the remaining valid elements based on the original size and partitioning method of the tensor), and then reads only the data within this valid range from external memory, ignoring invalid regions. After the transfer is complete, an additional marker is set for the boundary data block in the prefetch storage space, such as recording the valid data length of the block (e.g., 4 elements) in the metadata area of ​​the buffer. This marker can be a register value associated with the data block or stored in the header information of the buffer. In this way, invalid data transfer is avoided, bandwidth is saved, and correct data boundary information is provided for subsequent computation pipelines. It is worth noting that the special handling of boundary data blocks does not affect the monitoring of the data volume and the prefetch triggering logic.

[0087] When the elementwise vector computation pipeline reads data from the prefetch memory, for ordinary data blocks, the pipeline reads each element sequentially and performs computation according to the size of the complete data block (e.g., 32 elements) by default. However, for boundary data blocks with a valid data length marker, the pipeline first reads the marker before reading to determine the actual number of valid elements contained in the current boundary data block (e.g., 4). Subsequently, the pipeline only reads elements within this range and performs element-wise vector computation on them, ignoring data regions outside the marked length (these regions are not filled with valid data). For example, if the pipeline could normally process 8 elements per cycle, when processing this boundary data block, based on the valid data length of 4, only 4 element computations are performed, and then the data set is considered to have been completely consumed. By performing differentiated reading and computation based on the valid data length, this method can perfectly support tensors of arbitrary shapes without additional padding or pruning of the input data, solving the common boundary misalignment problem in practical applications. This allows the prefetching method of this invention to seamlessly handle tensor inputs of arbitrary sizes, further enhancing the practicality and robustness of the solution.

[0088] Following step S130, in step S140, the elementwise vector computation pipeline reads each set of data sequentially from the prefetch memory according to its inherent execution rhythm. Once the pipeline has completely read a set of data (e.g., all elements within a tile) and completed the corresponding element-wise operations (such as addition, ReLU activation, pruning, etc.), this set of data is considered consumed. At this point, the space occupied by this set of data in the prefetch memory becomes free, available for subsequent data transfer to overwrite or reuse. Simultaneously, the data volume described in S120 needs to be updated immediately, decrementing the counter by 1 to reflect a reduction of one unit in the number of data sets to be consumed in the buffer. The computation pipeline executes consumption operations independently, and its progress is unaffected by external storage access latency; as long as data is always available in the buffer, the pipeline can run continuously.

[0089] After each set of data is moved (step S130) or consumed (step S140), a judgment logic is triggered: the currently monitored amount of data (e.g., the value of a counter) is compared with the total capacity of the prefetch storage space (i.e., the maximum number of data sets that can be accommodated). If the amount of data is less than the capacity, it means that there is at least one free space in the buffer. At this time, the decoupled prefetcher (or control unit) immediately issues an instruction to the data moving module to start moving the next set of data. Conversely, if the amount of data is equal to the capacity, it means that the buffer is full, so the moving is paused, and the free space is released after the computation pipeline consumes the data before resuming. This judgment and triggering process is completely completed automatically by the hardware and does not depend on the current execution position of the elementwise computation pipeline. By repeatedly executing the closed-loop control of monitoring-comparison-triggering-consuming-updating, the system can autonomously and continuously prefetch data from external storage into the chip based solely on the free space of the buffer, while ensuring that the buffer does not overflow, until all the data to be processed has been moved and calculated. This mechanism allows data transfer to be performed in advance, overlapping with the execution of the computing pipeline in time, thereby effectively eliminating memory access waits and improving overall performance.

[0090] In conjunction with the first aspect, the size of the prefetch storage space is dynamically configured based on the type of the currently executed elementwise operator or the size of the tensor to be computed.

[0091] Understandably, the size of the prefetch memory allocated in step S110 is not fixed but can be dynamically adjusted according to actual computational needs. On one hand, different elementwise operators have different computational characteristics and data access patterns. For example, simple arithmetic operations such as addition and multiplication have extremely low computational latency but are more sensitive to data supply. In this case, a larger prefetch memory is needed to accommodate more data sets to be consumed, thus providing a more sufficient buffer to mask memory access latency. On the other hand, for activation functions such as ReLU and Sigmoid, although the computation itself is simple, it may involve nonlinear operations, and the size of the prefetch memory can be adjusted appropriately.

[0092] On the other hand, the size of the tensor to be computed also directly affects the prefetching strategy. For large tensors (such as 1024×1024 feature maps), using a larger prefetch storage space can allow more tiles to be prefetched into the chip, improving the overlap between transport and computation; for small tensors, an excessively large prefetch space would waste on-chip resources and can be reduced accordingly.

[0093] In this embodiment, the dynamic configuration can be implemented as follows: before the NPU starts executing the elementwise computation task, the instruction decoder or microcontroller calculates the optimal prefetch depth (i.e., the maximum number of data sets that can be accommodated) based on the operator type and tensor dimension information, and writes it into the decoupling prefetcher through the configuration register. Subsequently, the decoupling prefetcher sets the actual upper limit of the prefetch storage space according to the configuration value, and adjusts the capacity threshold used for comparison in step S150 accordingly.

[0094] Through dynamic configuration, this method can flexibly switch between different computing tasks, ensuring high performance while avoiding the waste of on-chip buffer resources, further improving the adaptability and energy efficiency of the NPU post-processing unit.

[0095] In conjunction with the first aspect, there are multiple data transport modules; these multiple data transport modules work in parallel to support the simultaneous execution of multi-path elementwise vector computation pipelines.

[0096] Each data transfer module corresponds to one elementwise vector computation pipeline, and each pipeline is independently equipped with prefetch storage space and a hardware counter for monitoring data volume.

[0097] In some complex NPU application scenarios, multiple independent element-wise post-processing tasks may need to be executed simultaneously, such as performing element-wise addition and ReLU activation on multiple different feature maps. To fully utilize the parallel processing capabilities of the NPU, this embodiment supports configuring multiple data transfer modules (such as multiple DMA channels) and multiple element-wise vector computation pipelines. Each computation pipeline has its own independent set of hardware resources, including: a dedicated prefetch memory space (allocated from a general-purpose buffer), a dedicated hardware counter (used to monitor the amount of unconsumed data in that pipeline), and a corresponding data transfer module. The pipelines are completely decoupled and independently execute the prefetching and computation processes described in S110 to S150.

[0098] For example, the first pipeline processes element-wise addition of tensor A, with its corresponding data transport module moving data from address region 1 of external memory to the first prefetch space, and the data volume is tracked by the first counter. Simultaneously, the second pipeline processes ReLU activation of tensor B, with its corresponding data transport module independently moving data from address region 2 to the second prefetch space, and the data volume is tracked by the second counter. The triggering of each transport and computation is independent, based on the idle state of its respective buffer. Through this parallel architecture, the NPU post-processing unit can handle multiple element-wise computation tasks simultaneously, significantly increasing overall throughput without increasing the latency of individual tasks. Furthermore, independently setting the prefetch memory space and counters avoids resource contention and logical coupling between different tasks, simplifying hardware design and improving system scalability.

[0099] In conjunction with the first aspect, if the monitored data volume is equal to the capacity of the prefetch storage space, it is determined that the prefetch storage space is full, and the data transfer module suspends transfer until the elementwise vector calculation pipeline consumes data so that the data volume is less than the capacity, at which point transfer resumes.

[0100] In conjunction with the first aspect, the size of the data block is matched to the width of the data group processed in a single operation of the elementwise vector computation pipeline.

[0101] Specifically, when dividing the tensor into data tiles in step S131, the size of each data tile is not chosen arbitrarily, but is matched with the width of the data elements that the elementwise vector computation pipeline can process in parallel at one time.

[0102] The elementwise vector computation pipeline in an NPU typically employs a SIMD (Single Instruction Multiple Data) architecture, where a fixed number of data elements can be processed simultaneously each clock cycle, such as 8, 16, or 32 FP16 values. This parallel processing width is the width of the data set processed in a single operation.

[0103] To ensure efficient operation of the computational pipeline, the number of data elements contained in each data block should be an exact multiple of its width, or equal to its width (when using a single-block, single-processing strategy). For example, if the computational pipeline can process 8 elements per cycle, the size of each data block can be set to 8, 16, or 24 elements, etc., ensuring that the pipeline does not need to perform additional boundary checks or padding operations within the block when reading and processing a complete data block. When the data block size is equal to the single-processing width of the computational pipeline, a single read completes the entire computation of a data block, resulting in the simplest control logic. If the data block size is an integer multiple of the width, the pipeline only needs to call the same processing unit multiple times consecutively to complete the computation of the entire data block. This matching design avoids the additional overhead caused by misalignment between the data block size and the hardware processing granularity (such as some data being split into multiple processing cycles or pipeline bubbles), thereby maximizing the parallel throughput capability of the computational pipeline. At the same time, this matching relationship also facilitates the capacity planning of the prefetch storage space: the total capacity of the prefetch storage space can be expressed as the sum of several data groups with a single processing width, so that the hardware counter count value in step S120 directly corresponds to the number of groups to be processed, simplifying the comparison and judgment logic.

[0104] Therefore, matching the data block size with the single processing width of the computation pipeline is beneficial to improving the overall performance and hardware efficiency of this method.

[0105] In conjunction with the first aspect, step S120 includes:

[0106] S121, calculate the difference between the write pointer and the read pointer; where the write pointer points to the write position of the latest set of data in the prefetch memory space, and the read pointer points to the read position of the latest set of data consumed.

[0107] S122, the difference is used as the amount of data that has been prefetched in the prefetch storage space but has not yet been consumed by the elementwise vector computation pipeline.

[0108] In the specific implementation of step S120, in addition to using a hardware counter, this embodiment can also track the occupancy status of the prefetch memory space by managing write pointers and read pointers.

[0109] The prefetch memory is organized as a circular buffer (FIFO), where the write pointer always points to the next free position, i.e., the position where the most recently moved set of data is about to be written; the read pointer always points to the next position to be consumed, i.e., the position after the most recently consumed set of data. Each time the data moving module successfully moves a set of elementwise computation data from external memory and writes it to the prefetch memory, the write pointer automatically moves forward by one data set unit (possibly using modulo operation wrapping). Each time the elementwise vector computation pipeline completely reads and consumes a set of data, the read pointer also automatically moves forward by one data set unit.

[0110] Therefore, the difference between the write pointer and the read pointer (typically expressed as (write pointer - read pointer + buffer capacity) mod buffer capacity in a circular buffer) accurately reflects the number of data groups that have been written to but not yet read from the current buffer. This difference can be calculated in real-time by a hardware subtractor or dedicated logic circuitry, eliminating the need to maintain a separate counter. Essentially equivalent to a hardware counter scheme, it provides an alternative hardware implementation option.

[0111] After calculating the difference between the write pointer and the read pointer, the system directly uses this difference as the amount of prefetched but not yet consumed data required in step S120. This difference is physically identical to the count value of the hardware counter: when the buffer is empty, the write pointer equals the read pointer, and the difference is 0; for each set of data moved, the write pointer leads the read pointer by one position, and the difference increases by 1; for each set of data consumed, the read pointer catches up by one position, and the difference decreases by 1.

[0112] Thus, in the subsequent step S150, when determining whether there is free space, it is only necessary to compare the difference with the capacity of the prefetch storage space: if the difference is less than the capacity, there is free space, and the transfer continues; if the difference is equal to the capacity, the buffer is full, and the transfer is paused.

[0113] By using read-write pointer interpolation, this embodiment eliminates the need for a dedicated counter register. Instead, it indirectly obtains the amount of unconsumed data by utilizing the inherent pointer information managed by the buffer, reducing hardware overhead. This is particularly suitable for NPU designs where the prefetch memory itself already possesses read-write pointer management logic. This alternative solution offers the same functionality and control effect as the hardware counter solution, both achieving decoupling between data handling and computation execution.

[0114] Secondly, the present invention also provides a data prefetching apparatus for NPU post-processing, the apparatus being used to perform the method described above. (Combined with...) Figure 2 As shown, the device includes: an on-chip general-purpose buffer 10, a data volume monitoring unit 20, a data transfer module 30, an elementwise vector computation pipeline 40, and a control unit 50.

[0115] The on-chip general-purpose buffer 10 includes a prefetch memory space of a predetermined size. This prefetch memory space is specifically used to temporarily store elementwise computation data prefetched from external memory, and its size can be dynamically configured according to the type of the current elementwise operator or the size of the tensor to be computed. The data output port of the prefetch memory space is directly connected to the data read port of the elementwise vector computation pipeline, enabling the computation pipeline to read data from it with low latency without having to access external memory each time.

[0116] The data volume monitoring unit 20 is used to monitor in real time the amount of data prefetched in the prefetch memory but not yet consumed by the elementwise vector computation pipeline. This data volume is a dynamically changing value: it increases after each successful transfer of a set of data into the prefetch memory; it decreases after each complete consumption of a set of data by the computation pipeline. The data volume monitoring unit can be implemented using a hardware counter or a read / write pointer difference calculation circuit. Regardless of the implementation method, its output value always accurately reflects the current occupancy depth of the buffer, providing a basis for subsequent prefetch decisions.

[0117] The data transfer module 30 is connected to external memory and prefetch memory. This module is responsible for reading a set of elementwise computation data from external memory according to a predetermined data tile size and writing it to an empty location in the prefetch memory. After each transfer operation is completed, the data transfer module sends a "transfer complete" trigger signal to the data volume monitoring unit, causing the data volume monitoring unit to increment the current data volume by one. The data transfer module can be a DMA controller, supporting multiple channels to work in parallel to achieve independent prefetching of multiple computation pipelines.

[0118] The elementwise vector computation pipeline 40 is the core component in the NPU post-processing that performs element-wise vector operations. This pipeline reads a complete set of data (e.g., a tile) from the prefetch memory and performs elementwise operations such as addition, multiplication, ReLU activation, or pruning. Whenever a set of data is completely consumed, the pipeline sends a "consumption complete" trigger signal to the data volume monitoring unit, causing the monitoring unit to decrement the current data volume by one. The computation pipeline operates at its own pace, and its execution progress is unaffected by external memory access latency; as long as there is data in the prefetch memory, the pipeline can operate continuously.

[0119] The control unit 50 is connected to the data volume monitoring unit and the data transfer module. The control unit continuously receives the current data volume value from the data volume monitoring unit and compares it with the total capacity of the prefetch storage space (i.e., the maximum number of data sets it can hold). When the monitored data volume is less than the capacity of the prefetch storage space, the control unit determines that there is free space and immediately issues a transfer command to the data transfer module to start transferring the next set of data. Conversely, if the data volume equals the capacity, it determines that the buffer is full, and the control unit suspends issuing transfer commands until the data volume falls below the capacity again after the computation pipeline consumes data, at which point transfer resumes. This judgment and triggering process is entirely automated by hardware, independent of the current execution progress of the elementwise computation pipeline, thus decoupling data transfer from computation execution.

[0120] The aforementioned units work collaboratively to form a closed-loop control: the data transport module 30 continuously prefetches data according to the instructions of the control unit 50; the data volume monitoring unit 20 dynamically updates the amount of unconsumed data; the elementwise vector computation pipeline 40 independently consumes data; and the control unit 50 makes autonomous decisions based on the buffer's idle state. Through this architecture, external memory access can be initiated before the computation consumes the current data, allowing data transport and computation to fully overlap in time. This effectively avoids the computation pipeline from idling while waiting for data, improving the utilization of NPU post-processing computing resources and overall throughput performance.

[0121] Meanwhile, the device supports tile partitioning, boundary processing, dynamic configuration of prefetch space, and multi-path parallel expansion, exhibiting good flexibility and scalability.

[0122] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and apparatus described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0123] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0124] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0125] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0126] Finally, it should be noted that the above embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A data prefetching method for NPU post-processing, characterized in that, The method includes: In the NPU's on-chip general-purpose buffer, a prefetch memory space of a predetermined size is allocated for elementwise vector computation; Monitor the amount of data in the prefetch storage space that has been prefetched but not yet consumed by the elementwise vector computation pipeline. The amount of data increases after each successful transfer of a set of data and decreases after each complete consumption of a set of data. The data transfer module moves a set of elementwise computation data from external memory to the prefetch storage space, and updates the data volume to increase it after the transfer is completed. The elementwise vector computation pipeline reads and consumes a set of data completely from the prefetch storage space, performs element-wise vector computation, and updates the amount of data to reduce it after consumption is completed. If the monitored amount of data is less than the capacity of the prefetch storage space, it is determined that there is free space, and the next set of elementwise calculation data is moved from the external memory to the prefetch storage space again through the data transfer module, until all elementwise data to be calculated has been moved and consumed.

2. The method according to claim 1, characterized in that, The step of monitoring the amount of data prefetched but not yet consumed by the elementwise vector computation pipeline in the prefetch memory includes: The number of data sets that have been prefetched but not yet fully consumed in the prefetch storage space is recorded in real time by a hardware counter. After each successful transfer of a set of data, the count value of the hardware counter is incremented by one to update the data volume; After each complete consumption of a set of data, the count value of the hardware counter is decremented by one to update the data volume.

3. The method according to claim 1, characterized in that, The steps of transferring a set of elementwise computation data from external memory to the prefetch memory space via the data transfer module include: The tensor to be computed using elementwise methods is divided into multiple data blocks along at least one spatial dimension; The data of each data block is sequentially transferred from the external memory to the prefetch storage space through the data transfer module, wherein each data block constitutes a set of elementwise computation data.

4. The method according to claim 3, characterized in that, Also includes: For boundary data blocks located at tensor boundaries and whose data size is less than a complete data block, the data transport module only transports the valid data in the boundary data block and marks the valid data length of the boundary data block in the prefetch storage space. The elementwise vector computation pipeline reads and consumes the boundary data block from the prefetch storage space based on the effective data length.

5. The method according to claim 1, characterized in that, The size of the prefetch storage space is dynamically configured based on the type of the currently executed elementwise operator or the size of the tensor to be computed.

6. The method according to claim 1, characterized in that, There are multiple data transport modules; multiple data transport modules work in parallel to support the simultaneous execution of multiple elementwise vector computation pipelines. Each of the data transport modules corresponds to one elementwise vector calculation pipeline, and each pipeline is independently equipped with a prefetch storage space and a hardware counter for monitoring the amount of data.

7. The method according to claim 1, characterized in that, If the monitored data volume is equal to the capacity of the prefetch storage space, it is determined that the prefetch storage space is full, and the data transfer module pauses transfer until the elementwise vector calculation pipeline consumes data to make the data volume less than the capacity, at which point transfer resumes.

8. The method according to claim 3, characterized in that, The size of the data block is matched to the width of the data group processed in a single operation of the elementwise vector computation pipeline.

9. The method according to claim 1, characterized in that, The step of monitoring the amount of data prefetched but not yet consumed by the elementwise vector computation pipeline in the prefetch memory includes: Calculate the difference between the write pointer and the read pointer; wherein the write pointer points to the write position of the most recently moved set of data in the prefetch memory space, and the read pointer points to the read position of the most recently consumed set of data; The difference is used as the amount of data in the prefetch storage space that has been prefetched but not yet consumed by the elementwise vector computation pipeline.

10. A data prefetching device for NPU post-processing, characterized in that, The apparatus is configured to perform the method as described in any one of claims 1-9; the apparatus comprises: An on-chip general-purpose buffer is provided, in which a prefetch memory space of a predetermined size is allocated. The prefetch memory space is used to store elementwise computation data prefetched from external memory and is connected to the data read port of the elementwise vector computation pipeline. The data volume monitoring unit is used to monitor the amount of data in the prefetch storage space that has been prefetched but not yet consumed by the elementwise vector computation pipeline, wherein the data volume increases after each successful transfer of a set of data and decreases after each complete consumption of a set of data. The data transfer module is connected to the external memory and the prefetch storage space. It is used to transfer a set of elementwise computation data from the external memory to the prefetch storage space, and after the transfer is completed, it triggers the data volume monitoring unit to update the data volume to increase it. The elementwise vector computation pipeline is used to completely read and consume a set of data from the prefetch storage space, perform element-wise vector computation, and trigger the data volume monitoring unit to update the data volume to reduce it after consumption is completed. The control unit, connected to the data volume monitoring unit and the data transfer module, is used to determine that there is free space when the monitored data volume is less than the capacity of the prefetch storage space, and to control the data transfer module to transfer the next set of elementwise calculation data from the external memory to the prefetch storage space again, until all elementwise data to be calculated has been transferred and consumed.

Images