Deep learning inference acceleration method based on DNN operator parallelism

Abstract

This invention discloses a deep learning inference acceleration method based on DNN operator parallelism. The method includes a stream allocator for deep neural network (DNN) operators and a DNN operator emitter. Specifically, using a user-provided DNN model and input tensors (i.e., inference data), the stream allocator first determines the CUDA streams allocated to the operators based on the DNN model structure. Then, by collecting offline performance data generated by model inference, the DNN operator emitter further optimizes the operator emission order on the GPU using an operator scheduling algorithm based on resource requirements and interference awareness. Finally, the CUDA stream allocation scheme and operator emission order are combined to generate a parallelized CUDA graph, thereby achieving efficient DNN inference on the GPU.

Description

Technical Field

[0001] The present invention relates to the field of deep learning model computing acceleration technology, and in particular to a deep learning reasoning acceleration method based on DNN operator parallelism, which improves GPU resource utilization while reducing DNN reasoning latency. Background Art

[0002] Deep neural networks (DNNs) have achieved remarkable success in various business areas, including image processing, speech recognition, and virtual reality. DNN inference tasks are known to be extremely sensitive to latency. For example, for safety reasons, the inference latency requirements for models in autonomous driving scenarios are very strict (e.g., within 100 milliseconds). To meet such performance requirements, modern cloud data centers host thousands of GPUs to accelerate users' DNN inference. For example, Alibaba Cloud has over 6,000 GPUs, many of which are tasked with managing a large number of inference requests.

[0003] GPUs in cloud data centers are equipped with increasingly powerful computing capabilities, which often exceed the resource requirements of a single inference task, leading to underutilization and waste of hardware resources. To achieve target model accuracy with less computation, recent work has focused on replacing traditional large operators with several smaller multi-branch operators in DNN models, further exacerbating the underutilization of GPU resources. While batching requests or concurrent processing of multi-model inference tasks can alleviate the low utilization of GPU resources, the increase in batched data volume and performance interference between multiple models inevitably lead to increased model inference latency. Therefore, it is imperative to improve GPU utilization without compromising DNN inference latency. Since DNN models can often be represented by a directed acyclic graph (DAG) with parallel operators, this provides an opportunity to exploit operator parallelism to accelerate DNN inference on GPUs while improving GPU utilization.

[0004] Existing methods for operator parallelization mainly include those based on bipartite graph transformation and dynamic programming. The bipartite graph transformation method introduces significant computational overhead and fails to account for inter-operator interference. The dynamic programming method also requires lengthy searches and introduces excessive inter-device synchronization, resulting in performance degradation. Summary of the Invention

[0005] In order to solve the above problems, the purpose of the present invention is to provide a deep learning inference acceleration method based on DNN operator parallelism, that is, a parallel scheduling strategy based on the resource requirements of operators without dependencies in DAG and offline collection operators.

[0006] The specific technical solution for achieving the purpose of the present invention is:

[0007] A deep learning inference acceleration method based on DNN operator parallelism specifically includes the following steps:

[0008] Step 1: Submit the DNN model and input data;

[0009] Step 2: Generate a flow allocation plan based on the model structure;

[0010] Convert the DNN model to a computational graph mode and traverse the nodes in the computational graph in topological order. For each node, traverse all its parent nodes until a parent node is found that satisfies the current node as its first successor node. Then, set the current node's flow to be consistent with the flow of the parent node. If no parent node that satisfies the conditions is found, set the current node's flow to a newly created flow.

[0011] Step 3: Run the DNN once with a batch of data to obtain the GPU resource requirements of the operator during the run.

[0012] Use PyTorch's analyzer to obtain operator runtime information and save it as a JSON file. Then parse all function call timestamps from the JSON file. Utilizing the characteristic that the kernel function run start timestamp must be later than the kernel function start timestamp, we derive the mapping relationship between the operator and its runtime resource requirement information, thereby determining the operator's number of threads, shared memory size, and number of registers.

[0013] Step 4: Based on the resource requirements obtained in step 3 and the operator category, determine an interference-aware operator launch order that can reduce GPU idle time;

[0014] First, obtain all operators to be emitted with an in-degree of 0, and divide these operators into two queues according to their categories, namely memory-intensive and compute-intensive, respectively. The following steps are repeated until both queues are empty. That is, each time an operator is emitted, alternately select a non-empty queue from the two queues, and select the operator with the least operator resource requirement from the selected queue for emission. After emission, update the in-degree of the operator, and then add the operators with an in-degree of 0 to the memory-intensive and compute-intensive queues according to their categories.

[0015] Step 5: Based on the stream allocation scheme in step 2 and the operator launch order in step 4, the specific execution process is captured and a CUDA graph is generated that can execute operators in parallel on the GPU;

[0016] Set all streams obtained in step 2 to pending capture mode. At the same time, traverse all nodes in the computation graph, create synchronization events for each node, and create an event list to save the events that the node depends on. Add the synchronization events of all parent nodes of each node that are not on the same stream to the event list. According to the operator emission order in step 4, emit the operators to the streams set to pending capture mode. At runtime, control each operator to execute only after all dependent events in its event list are completed, thereby capturing the entire execution process and generating a CUDA graph that can execute operators in parallel on the GPU and does not need to be captured again for subsequent runs.

[0017] This invention solves the problems of insufficient model inference resource utilization and slow inference speed in GPUs. It schedules parallel operators in the DAG to different streams as much as possible, while reducing stream synchronization overhead and enabling parallel execution of operators. Furthermore, by pre-determining the resource requirements of operators, classifying and sorting them, and launching them sequentially from smallest to largest, it improves GPU resource utilization and reduces model inference latency. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A system architecture diagram for implementing the present invention;

[0019] Figure 2 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0020] To make the objectives, technical solutions, and advantages of the present invention more clear, the following detailed description of the embodiments of the present invention is provided in conjunction with the accompanying drawings. The present invention designs and implements an efficient operator parallel scheduling framework to improve resource utilization and reduce inference latency for DNN inference on a GPU computing platform.

[0021] like Figure 1 As shown, the process of implementing the present invention includes: the user submits a DNN model and input data, the stream distributor converts the DNN model into a DAG representation and generates a stream distribution plan based on it, the model data collector obtains the resource requirements of the operator, the operator emitter generates the operator emission order, and the graph capturer combines the above to generate a parallelized CUDAGraph. Figure 2 , the specific steps are:

[0022] The user submits the DNN model and input data, and the stream distributor converts the DNN model into a computational graph mode and traverses the nodes in the computational graph in topological order. For each node, all its parent nodes are traversed until a parent node is found that satisfies the current node as its first successor node, and then the stream of the current node is set to be consistent with the stream of the parent node. If no parent node that meets the conditions is found, the stream of the current node is set to a newly created stream.

[0023] The model data collector determines whether the input data shape has been previously submitted. If so, the previous data information is reused. Otherwise, the DNN is run once, and the PyTorch analyzer is used to obtain the operator runtime information and save it as a JSON file. All function call timestamps are parsed from the JSON file. Taking advantage of the fact that the kernel function run start timestamp is always later than the kernel function start timestamp, a mapping relationship is derived between the operator and its runtime resource requirements, thereby determining the operator's number of threads, shared memory size, and number of registers.

[0024] The operator transmitter first obtains all operators to be emitted with an in-degree of 0, and divides these operators to be emitted into two queues according to the categories of the operators (memory-intensive and computation-intensive), namely the memory access queue and the computation queue; the following steps are executed in a loop until both queues are empty, that is, each time an operator is emitted, the non-empty queues in the two queues are alternately selected, and the operator with the least operator resource requirement is selected from the selected queue for emission; after emission, the in-degree of the operator is updated, and then the operators with an in-degree of 0 are added to the two queues according to their categories.

[0025] The graph capturer sets all streams obtained by the stream distributor to the pending capture mode, and traverses all nodes in the computational graph at the same time, creating synchronization events for each node and an event list to save the events that the node depends on. The synchronization events of all parent nodes of each node that are not on the same stream are added to the event list; then, according to the operator emission order of the operator emitter, the operators are emitted to the streams set to the pending capture mode. At runtime, each operator must be executed after all dependent events in its event list are completed, thereby capturing the entire execution process and generating a CUDAGraph that can execute operators in parallel on the GPU and does not need to be captured again for subsequent runs.

[0026] Based on the above functions, the system implements parallel scheduling of operators in the DNN model, while considering a better operator launch order to improve GPU resource utilization and reduce model inference latency.

[0027] Example

[0028] To verify the feasibility and superiority of the present invention, experiments were conducted on an NVIDIA A100-PCIe-40GB GPU equipped with an Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz and an NVIDIA GeForce RTX 2080 SUPER-8GB GPU equipped with an Intel(R) Core(TM) i9-10920X CPU @ 3.50GHz. PyTorch 2.0, CUDA 11.7, and cuDNN 8.5.0 were used. Classic DNN models were selected for the experiments, including Inception-v3, GoogLeNet, T5, and BERT.

[0029] Evaluation Benchmarks and Metrics: Opara, a scheduling framework implemented based on this invention, will be compared with the following three strategies: (1) native Pytorch, a native approach without CUDA Graph optimization; (2) sequential CUDA Graph, a sequential execution approach without operator parallelism; and (3) Nimble, which converts the computational graph into a bipartite graph and then identifies its maximum matching to determine the flow where each operator resides. The evaluation metrics are four key indicators: relative speedup of DNN inference latency, GPU utilization and GPU memory consumption, and runtime overhead.

[0030] In terms of inference latency, the relative speedup results of each method for different models are shown in Table 1.

[0031] Table 1 Relative speedup of each method for different models

[0032]

[0033] As can be seen in Table 1, Opara consistently outperforms the other three baseline methods in four classic DNN models. Specifically, compared to the existing PyTorch, Opara can achieve a minimum speedup of 1.85 times to a maximum of 4.18 times. This is because Opara leverages CUDA Graph to eliminate operator emission and function call overhead. Opara surpasses the sequential CUDA Graph by up to 1.68 times because Opara executes operators in the DNN in parallel. In addition, Opara outperforms Nimble by 1.29 times because it wisely alternates the scheduling of different types of operators with the lowest GPU resource consumption when launching each operator. In addition, Opara launches enough streams to improve parallelism (for example, Opara creates 28 streams for GoogLeNet, while Nimble creates 4 streams), which maximizes the parallelism of the operators.

[0034] Table 2 GPU utilization of different models under each method

[0035]

[0036]

[0037] To further reveal the performance improvement of Opara, we continue to study the GPU utilization (i.e., the SM usage efficiency of the GPU) during the execution of the DNN model. As shown in Table 2, Opara shows similar improvements in GPU utilization compared to the other three baselines in the table. Specifically, because Opara reduces the scheduling overhead of the original PyTorch, Opara significantly improves GPU utilization compared to the original PyTorch. Compared with the default CUDA Graph, Opara improves the GPU utilization of Inception-v3, GoogLeNet, BERT, and T5 by 36%, 58%, 20%, and 19%, respectively. Compared with Nimble, Opara improves GPU utilization by 1.01 to 1.42 times, which is mainly attributed to two facts: (1) maximizing flow allocation in Opara can increase the opportunities for parallel execution of operators; (2) Opara optimizes the startup order of operators, which can further reduce GPU idle time.

[0038] Table 3 GPU memory consumption of different models under each method

[0039]

[0040] In addition, in terms of GPU memory usage, as shown in Table 3, the parallel execution of DNN operators increases the amount of data residing in the GPU memory at the same time, resulting in higher peak GPU memory consumption for Opara than for sequential execution (i.e., PyTorch and sequential CUDA Graph).

[0041] Table 4 Runtime overhead of Opara and Nimble

[0042]

[0043] Opara's runtime overhead also performs well. As shown in Table 4, Opara's algorithm computation time for Inceptionv3, GoogLeNet, T5, and BERT is 0.50ms, 0.27ms, 2.80ms, and 0.58ms, respectively. In comparison, the computation time for the four models in Nimble is 14.10ms, 5.80ms, 161.40ms, and 20.8ms, respectively, which is at least an order of magnitude longer than Opara's computation time. This is because Nimble needs to transform the graph and perform an exhaustive search in the bipartite graph. This process is very time-consuming, with a time complexity of O(n 3), where n is the number of operators in the model's DAG. In contrast, Opara's time complexity can be reduced to O(n). This is because the inner loop of Opara's flow allocation algorithm depends only on the maximum width of the computation graph, which is typically small (i.e., less than 20). As DNN models become increasingly complex in the future, the number of operators will grow exponentially. When n is large enough, the overhead of Nimble's algorithm becomes unacceptable.

[0044] The embodiments of the present invention can also provide a deep learning inference acceleration system based on DNN operator parallelism. The system includes four modules: a stream allocation module, which generates a stream allocation plan based on the model structure; a model data collection module, which runs the DNN once and obtains the operator's resource requirements for the GPU during operation; an operator emission module, which determines a reasonable operator emission order based on the obtained operator resource requirements and the operator category; and a graph capture module, which captures the final specific execution process based on the stream allocation plan and operator emission order, and generates a CUDA graph that can execute operators in parallel on the GPU. Users only need to submit the DNN model and input data to the operator parallel acceleration system, and the system will automatically complete the efficient parallel scheduling of operators. This scheduling can improve the resource utilization of inference tasks on the GPU and reduce the latency of model inference.

Claims

1. A deep learning inference acceleration method based on DNN operator parallelism, characterized in that: The specific steps include: Step 1: Submit the DNN model and input data; Step 2: Generate a flow allocation plan based on the model structure; Step 3: Run the DNN once with a batch of data to obtain the GPU resource requirements of the operator during the run. Step 4: Based on the resource requirements obtained in step 3 and the operator category, determine an interference-aware operator launch order that can reduce GPU idle time; Step 5: Based on the stream allocation scheme in step 2 and the operator launch order in step 4, the specific execution process is captured to generate a CUDA graph that can execute operators in parallel on the GPU; where: Step 2 specifically includes: converting the DNN model into a computational graph mode, traversing the nodes in the computational graph in a topologically sorted order; for each node, traversing all its parent nodes until a parent node is found that satisfies the current node as its first successor node, and then setting the flow of the current node to be consistent with the flow of the parent node; if no parent node that meets the conditions is found, setting the flow of the current node to a newly created flow; The step 4 specifically includes: first obtaining all operators to be emitted with an in-degree of 0, and dividing these operators to be emitted into two queues according to the categories to which the operators belong, namely memory-intensive and computation-intensive, namely the memory access queue and the computation queue; looping through the following steps until both queues are empty, namely alternately selecting a non-empty queue from the two queues each time an operator is emitted, and selecting the operator with the least operator resource requirement from the selected queue for emission; updating the in-degree of the operator after emission, and then adding the operators with an in-degree of 0 to the memory-intensive and computation-intensive queues respectively according to their categories.

2. The deep learning inference acceleration method according to claim 1, characterized in that: Step 3 obtains the operator's GPU resource requirements during operation, including: using the PyTorch analyzer to obtain the operator's runtime information and save it as a JSON format file, parsing all function call timestamps from the JSON format file; using the characteristic that the kernel function running start timestamp must be later than the kernel function startup timestamp, deriving the mapping relationship between the operator and its runtime resource requirement information, thereby determining the operator's number of threads, shared memory size, and number of registers.

3. The deep learning inference acceleration method according to claim 1, characterized in that Step 5 generates a CUDA graph capable of executing operators in parallel on the GPU, including: setting all streams obtained in step 2 to a pending capture mode, traversing all nodes in the computation graph, creating synchronization events for each node and an event list to store the events that the node depends on, and adding the synchronization events of all parent nodes of each node that are not on the same stream to the event list; emitting operators to the streams set to pending capture mode according to the operator emission order in step 4, and controlling each operator at runtime to execute only after all dependent events in its event list are completed, thereby capturing the entire execution process and generating a CUDA graph that can execute operators in parallel on the GPU and does not require re-capture for subsequent runs.

Images