A method for batch-based neural network inference pipeline multiplexing

Abstract

This invention discloses a method for reusing a batch-processing-based neural network inference pipeline. The method includes the following steps: analyzing different stages of the network; segmenting each computational layer of the network into slice units; generating corresponding instructions based on the slice units; analyzing the memory access / computation ratio at different stages of the network to obtain a subgraph segmentation and fusion scheme; rearranging the instruction sequence of the matched subgraph using a dynamic programming algorithm; and deploying the subgraphs on the target hardware according to the instruction order. This invention mixes memory-intensive and computation-intensive operators by overlapping tasks, achieving a balanced load on memory access and improving hardware resource utilization without increasing hardware resources, thereby increasing computational throughput. Furthermore, this invention exhibits excellent portability and scalability.

Description

Technical Field

[0001] This invention relates to the field of neural networks, and also to NPUs, FPGAs, hardware and software co-optimization systems, neural network compilation, and in particular to a method for batch processing-based neural network inference pipeline multiplexing. Background Technology

[0002] Deep learning has been widely applied in fields such as image recognition, recommendation systems, and autonomous driving. However, with the widespread deployment of deep learning and neural networks in production and daily life, new challenges have been posed to the computing power of both cloud and edge devices. Traditional CPU architectures cannot meet the massive parallel computing demands of neural networks, leading to the widespread use of GPUs with general-purpose parallel computing architectures like CUDA for neural network training and inference. As networks evolve, their computational load increases further, and their computational structures become more complex. This widespread application has driven the evolution of hardware architectures, such as the introduction of Tensor Cores in Nvidia's V100 architecture, Google's TPU specifically designed for the TensorFlow framework, and companies like Cambricon and Huawei focusing on dedicated AI hardware.

[0003] A significant portion of high-performance computing architecture optimization focuses on computational tasks and hardware memory access analysis. By analyzing computational tasks, the bottleneck is identified as either memory bandwidth or computational components, and this information is used as a basis for architectural evolution. However, once the hardware is designed, its computational and memory resources, memory access paths, etc., are fixed. Yet, the demand for memory and computational resources changes at different stages of computation, leading to wasted computational or memory resources. This problem is difficult to solve through hardware design alone.

[0004] Analysis of the convolutional neural network (CNN) running on the currently used FPGA accelerator revealed that the FPGA-deployed accelerator exhibits alternating memory-intensive and computation-intensive operators at different stages of CNN operation. In the early layers of the CNN, the feature maps are typically large, while the input and output channels are usually small, resulting in a small amount of weight data and making it a computationally intensive operator. However, in the later stages of the network, the input and output channels expand rapidly, causing the amount of weight data to increase rapidly, thus transforming it into a memory-intensive operator.

[0005] Executing computationally intensive operators on a network can lead to wasted memory access bandwidth, while executing memory-intensive operators can result in idle computing components. Furthermore, data dependencies exist between computational layers within the same task, making sequential scheduling within a single task difficult. Real-world applications often involve continuous neural network processing. This invention leverages this characteristic to design a pipelined reuse mechanism based on batch processing of multiple parallel tasks. After dividing operators into instructions, the latter half of the memory-intensive operators from the previous task is mixed with the first half of the computationally intensive operators from the next task. This improves the utilization of memory and computing resources on the accelerator, thereby enhancing performance. Summary of the Invention

[0006] This invention proposes a batch-processing-based method for pipelined reuse of neural network inference, and an instruction scheduling method for accelerating neural network inference hardware. The technical solution of this invention addresses the waste of memory and computational resources caused by uneven memory access and computational loads at different stages of computation.

[0007] To solve the above problems, the present invention is achieved through the following technical solution:

[0008] A method for reusing a batch-processing-based neural network inference pipeline, characterized by the following steps:

[0009] Step 1) Analyze different stages of the network to obtain the access-to-compute ratio for each stage;

[0010] Step 2) Divide the network into slice units of similar size that are suitable for hardware execution.

[0011] Step 3) Generate corresponding instructions based on the slicing units in Step 2);

[0012] Step 4) Obtain the subgraph segmentation and fusion scheme by analyzing the memory access ratio at different stages of the network;

[0013] Step 5) The dynamic programming algorithm rearranges the instruction sequence of the matched subgraph;

[0014] Step 6) Deploy the equipment on the target hardware in the order of the instructions.

[0015] The aforementioned method for reusing a batch-processing-based neural network inference pipeline is characterized by the need to continuously process neural network inference tasks and perform pipelined scheduling on them.

[0016] The above-mentioned method for reusing neural network inference pipelines based on batch processing is characterized in that the subgraph segmentation and fusion scheme in step 4) is based on the analysis of the overall network's memory access and computation ratio, ensuring that the two subgraphs to be sequenced and fused are respectively computationally intensive and memory-intensive.

[0017] The above-described method for reusing a batch-processing-based neural network inference pipeline is characterized in that, in step 5), the dynamic programming algorithm uses the memory access ratio of the currently sorted instructions as the basis for selecting the next instruction.

[0018] The aforementioned method for reusing a batch-processing-based neural network inference pipeline is characterized in that the final instruction sequence obtained by this method forms a pipeline arrangement in which the preceding and following tasks partially overlap from the perspective of the task.

[0019] By adopting the above technical solutions, the present invention can achieve the following beneficial effects:

[0020] This invention addresses the problem of unbalanced computation-memory access load at different stages of operation in existing neural network inference accelerators. It proposes a pipelined reuse method based on batch processing, which mixes memory-intensive and computation-intensive operators by overlapping the arrangement of tasks before and after, thereby balancing the computation and memory access load. This improves the utilization of hardware resources without increasing hardware resources, thus increasing the throughput of computation. At the same time, the architecture of this invention has excellent portability and scalability. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the system architecture of the existing VTA deep learning accelerator.

[0022] Figure 2 This is a schematic diagram of the hardware architecture of the accelerator deployed in an embodiment of the present invention.

[0023] Figure 3 This is a schematic diagram of the acceleration hardware structure according to an embodiment of the present invention.

[0024] Figure 4 This is a schematic diagram of the subgraph division and subgraph fusion results in an embodiment of the present invention.

[0025] Figure 5 This is a flowchart of instruction sequence scheduling according to an embodiment of the present invention.

[0026] Figure 6 This is a diagram illustrating the effect of using 24 instructions as a test in an embodiment of the present invention.

[0027] Figure 7 This is a diagram showing the effect of using 16 instructions as a test in an embodiment of the present invention. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0029]

Example 1

[0030] like Figure 1 The illustration shows a prior art embodiment. The prior art VTA is an open-source deep learning accelerator, a general-purpose neural network inference accelerator driven by instructions. Figure 1 This is a schematic diagram of the system architecture of the VTA deep learning accelerator, and its execution flow is as follows:

[0031] Step 1) The instruction fetching unit reads and decodes the instruction from the host's memory, breaks the instruction down into three parts: fetching data, calculating, and writing back, and distributes them to the three execution units.

[0032] Step 2: The data retrieval unit retrieves the data required for calculation from memory according to the instructions, stores it in the input buffer and weight buffer, and passes a token to the subsequent calculation unit.

[0033] Step 3) After the calculation unit receives the token, it means that the data required for the calculation has been stored in the buffer and can be retrieved for calculation. After the calculation is completed, the data is stored in the output buffer and a token is passed to the subsequent write-back unit.

[0034] Step 4) After receiving the token, the write-back component indicates that the data in the output buffer is valid and stores the data back into memory.

[0035] Repeat steps 1) to 4) to execute the instructions one by one.

[0036] The design of VTA decoupled execution units can help with the early launch of memory access, which can alleviate the local memory access-computation load imbalance to some extent. However, from the perspective of the entire task, the memory access-computation load imbalance still exists.

[0037] Taking 8-bit quantized ResNet-18 as an example, most of its computations are convolution operations. The data volume and computational cost of each convolutional layer are shown in Table 1:

[0038]

[0039]

[0040] Table 1

[0041] The "64 3 224 224 7 2 3" indicates that the input layer has 64 input channels, 3 output channels, and an input of 224*224 two-dimensional tensor. The convolution kernel size is 7*7, the stride is 2, and the pad is 3.

[0042] The layers with a large computational load in the table are bolded. It can be seen that the memory access requirements of the later convolutional layers increase rapidly. In a fixed hardware design, when the bandwidth-computing power ratio is close to the memory-computing ratio of the overall network, computation-intensive or memory-intensive operations will occur at different stages of the operation. When computation-intensive or memory-intensive operators occur continuously within a long time window, and the buffer size is insufficient to meet the demand, it will lead to a waste of computing resources or memory access resources.

[0043] When VTA's existing solutions are used to execute such networks, the unbalanced load between memory access and computation leads to alternating waste of computational and memory access resources, which is a common phenomenon in current general-purpose edge device neural network inference accelerators.

[0044]

Example 2

[0045] The application scenario of this invention is the continuous processing of different inputs to the same neural network task, and on this basis, pipeline-style instruction rearrangement is performed on adjacent tasks.

[0046] To support this invention, the deployed acceleration hardware needs to meet the following three requirements:

[0047] 1. Supports flexible data prefetching operations;

[0048] 2. Controllable on-chip buffer space;

[0049] 3. A robust mechanism for separating memory access from computation.

[0050] Flexible data prefetching operations enable accelerators to perform prefetching actions when there are spare memory access resources, providing data for subsequent computations. Controllable on-chip buffer space ensures that data previously aggressively prefetched will not become invalid due to replacement strategies in traditional cache algorithms. A good memory access and computation separation mechanism can separate the memory access and computation of tasks, enabling higher utilization of memory access and computation components.

[0051] Taking the accelerator deployed in this invention as an example:

[0052] The hardware architecture of the accelerator deployed in this invention is as follows: Figure 2As shown, the architecture is mainly divided into several components: a control module, an instruction fetch module, an instruction dispatch module, a data fetch module, a computation module, a write-back module, and on-chip memory. The control module connects to the host via an AXI-Lite interface. The host writes values ​​to the control registers in the control module to record the starting address of the fetched instruction, the number of program instructions, accelerator activation, and performance monitoring information. The instruction fetch module is mainly used for instruction fetching. Based on the address in the control module, it retrieves the instruction from memory and dispatches it to the subsequent dispatch module. The dispatch module is mainly used for instruction decoding and dispatching. It distinguishes the type of the current instruction and dispatches it to the corresponding execution unit. Each type of unit has an instruction queue. When the dispatch condition of the instruction at the top of the queue is met and the execution unit is idle, the instruction can be dispatched, thereby reducing dummy operations in the pipeline. The data fetch module uses... An AXI interface connects to the host's memory. When a data retrieval instruction is received, a two-dimensional tensor is retrieved from memory via the AXI interface and stored in a block of on-chip memory. The on-chip memory component consists of several blocks, each capable of storing one tensor and can be occupied by one execution unit at a time. The on-chip memory component also has a scoreboard for fine-grained control of the on-chip memory. There are two types of computation modules: one for calculating instructions requiring numerous multiply-accumulate units, such as matrix multiplication and convolution, and the other for calculating element-wise operations such as addition, ReLU, and quantization. The write-back module, corresponding to the data retrieval module, can store the tensor from a block into a location in memory.

[0053] In the accelerator deployed in this invention, since the execution components are decoupled through a scoreboard and a vector bank, corresponding instructions are also designed to adapt to the hardware structure design. The instructions are divided into three categories: fetch, write back, and compute, which correspond to the three components respectively. Similar to the RISC instruction set, fetch is only responsible for reading tensors, write back is only responsible for writing back tensors, and compute is only responsible for computed tensors.

[0054] To adapt to batch-based neural network inference pipeline reuse methods, the accelerator deployed in this invention decouples the data fetching, computation, memory access components and instructions, enabling the accelerator to automatically perform aggressive prefetching, while the on-chip memory component controlled by the scoreboard can perform fine control over the on-chip memory.

[0055] This invention serves as an abstract structure for neural network inference hardware capable of deploying this invention, such as... Figure 3As shown, the hardware deployment of this invention is mainly composed of three parts: memory access path, on-chip memory, and computing cores. The memory access path automatically prefetches the data needed for subsequent computations until the on-chip memory is full. The on-chip memory acts as a buffer, prefetching data when memory access resources are abundant until the buffer is full, and providing data when computing resources are abundant until the prefetched data in the buffer is exhausted, thereby improving the utilization of the computing cores. The buffer can handle load imbalances within a small time window; however, performance is still affected by load imbalances across the entire task. The following explanation will be based on this abstract structure.

[0056] Instruction compilation and scheduling:

[0057] In the computational deployment of edge device inference acceleration hardware for neural networks, due to the limited on-chip storage size, it is necessary to load inputs and weights from host memory onto the chip in real time during computation. For computational flexibility, the computational tasks of each layer need to be divided according to their shape and then loaded onto the chip one by one for computation.

[0058] Neural network computations typically proceed linearly layer by layer, with the output of one layer serving as the input to the next. Similarly, neural network compilation follows a layer-by-layer approach, transforming or breaking down each computational layer into several operators. Each operator is then compiled as instructions, arranged sequentially, and data reuse between adjacent instructions is utilized. Taking a convolutional neural network (CNN) as an example, its core mainly consists of convolutional layers, pooling layers, and fully connected layers. With a batch size of 1, the main parameters of a convolutional layer include the number of input channels (I), the number of output channels (O), the feature map size (HW), and the convolutional window size (K). Dedicated acceleration hardware supports some general-purpose computations of fixed sizes, such as matrix multiplication. To adapt to relatively fixed hardware structures, when deploying convolutional layers, the compiler divides the computational task into smaller chunks across the various dimensions of the convolutional layer, corresponding to the hardware implementation in the accelerator. The partitioned computational tasks exhibit varying degrees of data reuse. For instance, input data is reused along the OK dimension, weights along the HW dimension, and output data along the IK dimension. Different convolutional layer shapes result in varying degrees of data reuse, leading to changes in the computation-to-memory access ratio. The partitioning and compilation operations for other pooling layers and fully connected layers are similar.

[0059] After dividing the computational layers in the neural network according to the hardware structure, due to the data dependency between the layers, the instruction sequences of each computational layer are arranged in the order of the computational layers. The instructions in a computational layer may use the same data. Arranging them together can take advantage of the spatial locality and temporal locality of data, make full use of data reuse in the computational layer, and reduce memory access requirements.

[0060] Then, based on the generated instructions, the overall memory access and computation requirements of the task are calculated and the average value is obtained.

[0061] Based on this average, computational layers with a value greater than this average are called memory-intensive, and those with a value less than this average are called computation-intensive. Several consecutive memory-intensive or computation-intensive computational layers are grouped into a subgraph. Then, sequentially, a subgraph is selected from the first subgraph. Starting from the middle of the task, another subgraph with the opposite memory-intensive computation ratio is selected, ensuring that the memory-intensive computation ratio after merging the two subgraphs is as close as possible to the average of the overall network. This process continues, selecting a subgraph sequentially, and then selecting another subgraph with the opposite memory-intensive computation ratio from the merged subgraph to merge, until the entire task is merged.

[0062] like Figure 4 As shown, Figure 4 The Central Plains task is divided into four subgraphs based on the memory access to computation ratio, with the overall memory access to computation ratio being 1. The first subgraph is selected from the front. Figure 1 Its memory access / computation ratio is 1 / 2, making it computationally intensive. Starting from the middle, a relatively small sub-array is selected. Figure 3 The memory access computation ratio is 2, which is memory-intensive. They are paired up, and then the child is selected in sequence. Figure 2 ,son Figure 2 It is memory-intensive; select a relatively compute-intensive sub-sub from the back. Figure 4 As a fusion object, after pairing, the ratio of statistical computation to memory access is more balanced compared to the previous structure.

[0063] Then we need to merge each pair of subgraphs, which is the instruction rearrangement operation.

[0064] Figure 5 This is a flowchart illustrating the instruction order scheduling process according to an embodiment of the present invention. It describes the scheduling process from graph partitioning to instruction order in the network, using dynamic programming to determine the order of instructions after the fusion of corresponding subgraphs.

[0065] 51) Calculate the average memory access ratio of the two subgraphs;

[0066] 52) Select one or more instructions from the first subgraph and move them to the tail of the instruction queue X;

[0067] 53) Calculate the memory access / computation ratio of the previously selected N instructions (N can be modified according to the effect);

[0068] 54) If the memory access / computation ratio calculated in step 53) is less than the average or the instructions of the computationally intensive subgraph are exhausted, select one or more instructions from the instruction queue of the memory-intensive subgraph and add them to the tail of the instruction queue X; if the total memory access / computation ratio of the current instructions is greater than or equal to the average or the instructions of the memory-intensive subgraph are exhausted, select one or more instructions from the instruction queue of the computationally intensive subgraph and add them to the tail of the instruction queue X.

[0069] 55) If there are still instructions in the instruction queues of both subgraphs, then jump to step 53);

[0070] 56) The instruction queue X is the merged instruction queue.

[0071] Additionally, if the hardware design has a relatively accurate performance simulator, modifications can be made to the integrated dynamic programming algorithm. The predetermined instruction sequence can be sent to the performance simulator, which can then return the bandwidth and computing power usage of the most recent execution time. If the computing units are full but the memory access units are relatively idle, it means that the recent execution used more computing units, so instructions are selected from the memory-intensive subgraph; otherwise, instructions are selected from the computation-intensive subgraph.

[0072] Next, we will use two examples to illustrate the actual effect of the present invention. The deployed hardware structure is the simple abstract model shown earlier. The hardware will automatically perform prefetching operations, fetch the data needed by the subsequent instructions into the on-chip buffer, and release the data used by the computed instructions. The subsequent computation can also automatically find the data in the on-chip buffer. We will analyze the memory access time axis and the computation time axis.

[0073]

Example 3

[0074] like Figure 6 As shown, 24 instructions were used as the test run. It is assumed that the execution time of each instruction is 2, the data size required for the first 12 instructions is 1, the data size required for the last 12 instructions is 3, the bandwidth is 2, and the on-chip buffer size is 6.

[0075] If we assume that this invention is not used and these 24 instructions are deployed on the hardware, we can see that in the first half of the operation, after the buffer is full, there are many idle times on the memory access timeline. Later, when memory access resources are insufficient, there are many idle times on the computation timeline. From the perspective of the overall task, there is a waste of both memory access resources and computation resources, resulting in an overall performance decline. The total time spent is 29.

[0076] If a batch-based pipelined multiplexing method is used, the first 12 instructions are treated as the first subgraph, and the last 12 instructions as the second subgraph. These are then merged, resulting in an overall average memory access / computation ratio of 2. The first subgraph (12 instructions) is computationally intensive, while the second subgraph is memory-intensive, based on the previously described method:

[0077] A command is fetched from the first subgraph, with a memory access / computation value of 1, which is less than the overall average.

[0078] A command is retrieved from the second subgraph, with a memory access / computation value of 2, which is equal to the overall average.

[0079] Retrieving an instruction from the first subgraph results in a memory access / computation ratio of 5 / 6, which is less than the overall average.

[0080] ...

[0081] The instructions for the two subgraphs will be arranged in a "1212..." pattern. During data calculations, such as... Figure 6 As shown, the idle time of previous memory accesses and computations is filled by each other, and the relatively balanced load design reduces the pressure on the buffer, resulting in an overall performance improvement. The total time spent is approximately 24.5 seconds.

[0082]

Example 4

[0083] like Figure 7 As shown, 16 instructions are used for the test. It is assumed that the computation time of the first 8 instructions is 1 and the data size required is 1, the computation time of instructions 9-12 is 1 and the data size required is 2, the computation time of the last 4 instructions is 1 and the data size required is 4, the on-chip buffer size is 6, and the bandwidth is 2.

[0084] Without this invention, the first 8 instructions are computationally intensive, with the buffer constantly prefetching subsequent instructions. When the buffer is full, memory access resources are wasted. Instructions 9-12 are computationally and memory-intensive, with the buffer occupancy rate remaining basically unchanged. The last 4 instructions are memory-intensive, with the prefetched data in the buffer being continuously consumed. When the buffer reaches 0, computational resources are wasted, with an average time of 19.5 seconds per task.

[0085] Using this invention to schedule instructions, the first 8 instructions are used as the first subgraph, instructions 9-12 as the second subgraph, and the last 4 instructions as the third subgraph. The first and third subgraphs are merged. According to the previously described method, the average memory access / computation ratio of the merged subgraph is 2.

[0086] From son Figure 1 A single instruction is received, with a memory access / computation ratio of 1, which is less than the overall average.

[0087] From son Figure 3When one instruction is received, the memory access / computation ratio is 5 / 4, which is greater than the overall average.

[0088] From son Figure 1 When an instruction is fetched, the memory access / computation ratio is 2, which is equal to the overall average.

[0089] From son Figure 1 When one instruction is received, the memory access / computation ratio is 7 / 8, which is less than the overall average.

[0090] ...

[0091] Instructions are scheduled in the format of "131131...", and the relatively balanced memory access-computation load greatly reduces the pressure on the buffer, with an average time of 17 seconds per task.

[0092] The key technical points and innovations of this invention are as follows:

[0093] 1. Employ batch processing-based balanced scheduling for both computationally intensive and memory-intensive tasks to balance the scheduling of different modules of multiple tasks.

[0094] 2. By overlapping compute-intensive and memory-intensive tasks on the timeline to form a pipeline, the system performance can be improved.

[0095] 3. The hardware design needs to support flexible data prefetching operations, controllable on-chip buffer space, and a good memory access and computation separation mechanism, without requiring additional computing cores and bandwidth;

[0096] 4. Instruction scheduling adopts a dynamic programming algorithm based on the module memory access ratio for module scheduling.

[0097] The above description is merely a preferred embodiment of the present invention, but the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention. Any simple modifications, equivalent changes, and alterations made by those skilled in the art to the above embodiments based on the technical essence of the present invention without departing from the principles and spirit of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for batch-based neural network inference pipeline multiplexing, the method comprising: The method comprises the following steps: Step 1) Analyzing different stages of the network to obtain the memory-computation ratio of each stage; Step 2) Splitting each computing layer of the network to obtain slice units of similar size suitable for hardware execution; Step 3) Generating corresponding instructions according to the slice units in step 2); Step 4) Obtaining a subgraph splitting and fusion scheme by analyzing the memory-computation ratio of different stages of the network; Step 5) Rearranging the instruction sequence of the matched subgraph by a dynamic programming algorithm; the instruction sequence obtained in step 5) forms a pipeline arrangement in which the tasks partially overlap from the perspective of the tasks; Step 6) Deploying on the target hardware according to the instruction sequence.

2. The method for batch-based neural network inference pipeline multiplexing according to claim 1, wherein, The inference task of the neural network needs to be processed continuously, and it is scheduled in a pipeline manner.

3. The method for batch-based neural network inference pipeline multiplexing according to claim 1, wherein, The subgraph splitting and fusion scheme in step 4) is based on the analysis of the memory-computation ratio of the overall network, and the two subgraphs that are guaranteed to be sequential and fused are computation-intensive and memory-intensive.

4. The method for batch-based neural network inference pipeline multiplexing according to claim 1, wherein, In step 5), the dynamic programming algorithm uses the memory-computation ratio of the currently sorted and completed instructions as the basis for selecting the next instructions.

Images