GPU scheduling method and system based on asynchronous data transmission

Abstract

The invention provides a GPU scheduling method and system based on asynchronous data transmission. During deep learning reasoning, data transmission from the CPU to the GPU and GPU calculation are executed asynchronously, and the final delay time can be greatly shortened. Therefore, the invention provides a quantitative model taking the concurrency as an independent variable and taking the systemthroughput and the time delay as dependent variables. Based on the model, a scheduling algorithm for hiding data transmission delay by utilizing two processes is realized, so that the system performance is improved. According to the method and system, the next batch size can be calculated and determined through the batch job information which is being executed, and GPU data transmission and calculation processes are completely parallel. Meanwhile, the algorithm can match continuously changing concurrency in real time, so that the operation delay is reduced to the maximum extent while the real-time throughput requirement is met.

Description

technical field [0001] The present invention relates to the technical field of GPU scheduling, and in particular to a GPU scheduling method and system based on asynchronous data transmission. Background technique [0002] Deep learning is divided into two processes: training and reasoning. Reasoning is the application of the ability learned by deep learning during training to practice. Therefore, in the actual production environment, more attention is paid to reasoning. In reality, due to the increasing demand for computing power of the deep learning model itself, as well as the rapid growth of the number of users and task submissions of deep learning applications on the product side, the accelerated optimization of deep learning reasoning systems has become one of the current research hotspots. [0003] In recent years, accelerated optimization methods for deep learning inference have begun to develop. Most of the current methods focus on using dedicated hardware to reduce...

AI Technical Summary

Problems solved by technology

In order to solve the high latency and low throughput problems faced by deep learning reasoning in the actual production environment, the present invention provides a scheduling method for deep learning reasoning systems, which utilizes concurrency, time delay, and batch size. Relationship, to predict the size of the next batch of jobs, in order to achieve the effect of hiding data transmission, so that the system can shorten the job delay time under the premise of meeting the throughput requirements

Images