A GPU Scheduling Optimization Method for Deep Learning Inference Service System

Abstract

The present invention discloses a GPU scheduling optimization method for a deep learning inference service system, including initializing the deep learning inference service system; allocating all models included in the processed system obtained, starting a prediction thread, periodically executing a throughput demand prediction process, and predicting the throughput demand of the models allocated to the prediction thread in a new period; starting a scheduling thread, and using the obtained predicted throughput demand to execute a throughput adjustment process based on a feedback control strategy at the scheduling moment to optimize the actual throughput allocation of each model in the system. The present invention can dynamically predict the throughput of each model service when exclusive GPUs are used, effectively adapt to complex and changeable workloads; meet the different latency and throughput requirements of model requests deployed on the same server, and make up for the deficiencies of task scheduling strategies in existing model service systems.

Description

Technical Field

[0001] The present invention belongs to the technical fields of computer system architecture and artificial intelligence, and particularly relates to a GPU scheduling optimization method for a deep learning inference service system. Background Art

[0002] With the explosive growth of data, the improvement of core algorithms, and the enhancement of hardware computing power, deep learning technology has brought inestimable value to the development and application of artificial intelligence and has been widely used in fields such as computer vision, speech recognition, and natural language processing. Deep learning mainly includes two stages: "training" and "inference". The training stage is a process of continuously adjusting the weights of a deep neural network (DNN) according to input data using an optimization algorithm, which is a process of building a model. Since the number of input data and DNN weight parameters is large, training these DNN models usually requires a large amount of computing power and takes several hours to several days to complete the training process.

[0003] Hardware accelerators such as GPUs (Graphics Processing Unit) and TPUs (Tensor Processing Unit) have become the mainstream hardware for accelerating DNN model training due to their powerful ability to perform simple repetitive operations on data. After the DNN model is trained, predictions can be made on the input data, and then it enters the inference stage. The online inference service of the DNN model is an important applicable scenario for deep learning technology. It deploys the DNN model in a cloud data center to provide inference services for tenants, enabling mobile devices with limited computing resources to also support deep learning applications through this service. In the early days, inference tasks were more often deployed on CPUs (Central Processing Unit). As DNN models grow larger, it is difficult to meet the end-to-end real-time requirements (latency less than 100 ms) when running inference tasks on CPUs. Therefore, currently, a large amount of work favors using GPUs to accelerate DNN inference tasks.

[0004] To improve the utilization rate of GPU resources, the current common method is to deploy multiple deep learning models on the same GPU server. In this scenario, requirements such as low latency of inference tasks and performance isolation among multiple tenants pose new challenges to the task scheduling mechanism in a shared GPU environment. Currently, task scheduling in a shared GPU environment is mainly divided into space-sharing and time-sharing types. Space-sharing technologies represented by NVIDIA's Multi-Process Service (MPS) can enable the GPU to run multiple tasks at any time, effectively improving the resource utilization rate of the inference service system. However, under the space-sharing strategy, the performance of simultaneously running tasks has great uncertainty, and it is impossible to guarantee the performance isolation and real-time requirements among multiple tenants. In any scheduling unit under the time-sharing strategy, the GPU only runs a single inference task, without fully utilizing the parallel execution characteristics of the GPU cores. Compared with the space-sharing strategy, although the GPU resource utilization rate is lower under the time-sharing strategy, the execution time of inference tasks is more stable, and the real-time requirements of inference service requests can be better guaranteed. However, when heterogeneous models are deployed on the same server, the service performance of relatively small-scale models is more likely to be interfered with, seriously affecting the service experience of the corresponding tenants.

[0005] In summary, for the acceleration process of DNN inference tasks, the current shared GPU environment cannot well schedule multiple tasks, has uncertainties and is prone to interference, and cannot well guarantee the real-time requirements of inference service requests.

[0006] Summary of the Invention

[0007] The purpose of the present invention is to propose a GPU scheduling optimization method for a deep learning inference service system, which is used to support a deep learning inference service system with multi-tenant isolation in a shared GPU environment. It can dynamically and real-time predict the throughput requirements of each model service in a shared GPU environment under complex and changeable workload conditions, and based on the prediction results, perform dynamic and real-time GPU scheduling to solve the performance isolation problem among multiple tenants.

[0008] The GPU scheduling optimization method for a deep learning inference service system provided by the present invention includes the following steps:

[0009] S1. Initialize the scheduling optimization parameters;

[0010] S2. Obtain the tasks to be processed by the deep learning inference service system at the current moment. Based on the system parameters of the deep learning inference service system, predict the throughput requirements of each model in the new round of the cycle for the tasks to be processed; where the model throughput refers to the number of inference requests successfully responded by the model; each inference request has a deadline, and only when the client obtains the corresponding inference result before the deadline of the request, the request is considered a successful response and is counted into the model's throughput.

[0011] S3. Start a new round of the cycle. Based on the throughput requirements of each model in the new round of the cycle obtained in step S2, adjust and allocate the throughput of each model within the cycle duration.

[0012] S4. After the current cycle ends, repeat steps S2 - S3 until the deep learning inference service system terminates operation, completing the GPU scheduling optimization for the deep learning inference service system.

[0013] The initialization of the scheduling optimization parameters described in step S1 specifically includes: The globally shared data defined by the system includes the model variable array models[2][m], the model state index si, and the model state lock variable si_lock corresponding to the model state index si. Set up a simulation queue and a request queue for each model; where models[2][m] is used to store the state information, and the stored state information includes the estimated value of the throughput requirement of each model and the actual throughput in the current cycle; m is the number of models, each column of the models[2][m] array corresponds to the state information of a model. Set the initial values of each element in models[2][m] and si to 0, and the value range of si is limited to 0 or 1; to avoid read - write conflicts of the model state information during the prediction process and the scheduling process, the scheduling process only reads and writes the model state stored in each column of the models[2][m] array from the row specified by the si value, and the prediction process reads and writes the model state stored in each column of the models[2][m] array from the row specified by the si′ value, where si′=(si + 1) % 2, and % is the modulo symbol; the number of parallel - executed prediction processes is n, and the number of parallel - executed scheduling processes is k. m, n, and k are all natural integers, and the value of m needs to satisfy being greater than or equal to 1. The specific values of n and k are set according to the hardware configuration of the system; the request queue of each model is used to store the inference requests waiting for response of the model, and the simulation queue is used to store the inference requests involved in the prediction process described in step S2.

[0014] The prediction of the throughput requirements of each model in the new round of the cycle based on the system parameters of the deep learning inference service system described in step S2 is specifically to predict the throughput requirements by the following steps:

[0015] A. Evenly distribute all models to each prediction process: The number of models in the system is m, and the number of prediction processes executed in parallel is n. Both m and n are natural numbers, where m is greater than or equal to 1, and the specific value of n is set according to the system's hardware configuration and is greater than or equal to 1. The number of models assigned to the first n - 1 prediction processes is ones, where

[0016] is the ceiling function, and the number of models assigned to the last prediction process is m % n, where % is the modulo operation;

[0017] B. Start the prediction process to estimate the throughput requirements of all models in a new cycle: For the current model, the prediction process assumes that the model exclusively occupies the GPU computing resources, simulates the scheduling of the requests that the model has not completed, calculates the number of inference requests that can be successfully responded to within one cycle duration under the simulated scheduling situation, and takes the calculation result as the throughput requirement of the model in the new cycle;

[0018] C. All prediction processes end after completing the estimation task;

[0019] D. Obtain the model status lock variable si_lock, update the model status index si to si = si', publish the model status data in the new cycle for the scheduling process, and release the model status lock variable after the update;

[0020] E. End this round of prediction process;

[0021] The detailed process for the prediction process to estimate the throughput requirement for a single model specifically includes:

[0022] (1) Copy all requests in the request queue of the current model to the simulation queue of the model;

[0023] (2) Clear the status data to be published for the current model: Assume the number of the current model is i, and the element corresponding to models[(si + 1) % 2][i] is the status data to be published for the model. Each model status data contains two member variables: sim_solo and goodput. Among them, models[(si + 1) % 2][i].sim_solo is the sim_solo member variable of the model numbered i, which is used to record the throughput requirement estimated for the current model, and models[(si + 1) % 2][i].goodput is the goodput member variable of the model numbered i, which is used to record the actual throughput of the current model. Assign the sim_solo and goodput member variables of each model status data to 0 to clear the status data to be published for the current model;

[0024] (4) Set the start and end variables, which are used to indicate the start time and end time of a new round of the cycle respectively; read the current system time and assign it to start; then assign the result of adding start and the cycle duration to end;

[0025] (5) Assume the system has N GPUs, set the simulation scheduling time sched_time for each GPU, and initialize sched_time to start;

[0026] (6) Determine whether the simulation queue is empty: if it is empty, execute step (13); otherwise, execute step (7);

[0027] (7) Obtain the minimum sched_time value among all GPUs and assign the result to the variable min_sched_time. min_sched_time is a temporary variable that records the minimum sched_time value among all GPUs; set the GPU with the minimum sched_time value as GPU i ;

[0028] (8) Determine whether min_sched_time is greater than or equal to end: if so, execute step (13); otherwise, execute step (9);

[0029] (9) Search for as many consecutive request blocks as possible in the simulation queue and meet the following requirements:

[0030] min_sched_time + inferTime(batch_size) < deadline

[0031] where the deadline for the first request in the request block is deadline, the number of requests in the block is batch_size; inferTime(batch_size) is the completion time for batch executing batch_size current model requests; the above formula is used to determine whether the request block can be batch executed on the GPU when the current model is exclusive to the GPU;

[0032] (10) Update the throughput of the current model in the simulation scheduling scenario to sim_solo + batch_size; where sim_solo is a member variable of the model status data;

[0033] (11) Update the simulation scheduling time sched_time of GPU i in the simulation scheduling scenario to min_sched_time + inferTime(batch_size);

[0034] (12) Delete all requests in the consecutive request block described in step (9) from the simulation queue, and jump to step (6);

[0035] (13) End the prediction process for the current model.

[0036] Adjust and allocate the throughput of each model in the new round of cycle based on the throughput requirements of each model obtained in step S2 in step S3, which specifically includes the following steps:

[0037] 1) Obtain the model status lock variable si_lock;

[0038] 2) Copy all elements in the row indicated by the model status index si in the variable array models[2][m] to the local one-dimensional array ms, and then release the model status lock variable si_lock;

[0039] 3) Traverse each element in the array ms, and judge whether the actual throughput obtained by each model is lower than the standard value: If Then the actual throughput obtained by this model is lower than the standard value; m i Is the (i + 1)-th element in the array ms;

[0040] 4) If the actual throughput of the model is lower than the standard value, add these models to the M list with an initial value of empty; otherwise, add all models to the M list;

[0041] 5) Find as many consecutive request blocks as possible from the request queues of each model to meet the following requirements:

[0042] curTime + inferTime(batch_size) < deadline

[0043] In the formula, the arrival time of the first request in the request block is arrival_time, the deadline is deadline, the number of requests in the block is batch_size, the current system time is curTime, and inferTime(batch_size) is the time to execute batch_size current model requests in batch; deadline - inferTime(batch_size) is the latest scheduling time when all requests in the request block can be successfully responded; then set the time for the next scheduling process for the current GPU to curTime + inferTime(batch_size);

[0044] 6) Exit the scheduling process.

[0045] The GPU scheduling optimization method for the deep learning inference service system provided by the present invention initializes the deep learning inference service system. In a shared GPU environment, based on the real-time load conditions allocated to each model, it can dynamically predict the throughput of each model service when using an exclusive GPU, and the predicted results can effectively adapt to complex and variable workloads without the need to introduce an additional offline prediction process. At the same time, taking the throughput when the model is placed alone as a metric, it adjusts the weights in terms of GPU resource allocation between heterogeneous models to meet the different latency and throughput requirements of model requests deployed on the same server, making up for the deficiency of the task scheduling strategy in the existing model service system in terms of performance isolation for heterogeneous model requests. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a GPU scheduling framework diagram for the deep learning inference service system.

[0047] Figure 2 It is a schematic flow chart of the method of the present invention.

[0048] Figure 3 It is a schematic overall flow chart of throughput requirement prediction.

[0049] Figure 4 It is a detailed flow chart of the prediction thread estimating the throughput requirement within a new round for a single model.

[0050] Figure 5 It is a schematic flow chart of throughput adjustment based on the feedback control strategy.

[0051] Figure 6 It is a schematic diagram comparing the performance of Embodiment 1 of the present invention with the prior art. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0052] Such as Figure 1The following is a GPU scheduling framework diagram for a deep learning inference service system: It mainly consists of two parts, namely, a throughput demand prediction module and a throughput adjustment module based on a feedback control strategy. The two parts interact with each other to complete the overall operation. The scheduling system maintains an inference request queue for each model and continuously receives inference requests from clients, inserting the inference requests into the inference request queue of the target model. In the scenario where multiple models share the GPU, the throughput demand prediction module periodically estimates the throughput of each model when it runs independently under the current workload conditions, and then based on the estimation results, determines the proportion of throughput demands among the models, providing guidance information to the throughput adjustment module based on the feedback control strategy. The throughput adjustment module based on the feedback control strategy dynamically monitors the actual throughput of each model in the current cycle, and based on the proportion of throughput demands among the models, finely optimizes the actual throughput allocation, minimizing the differences in performance losses caused by GPU sharing for heterogeneous models and ensuring performance isolation among the models.

[0053] As Figure 2 The following is a schematic diagram of the method flow of the method of the present invention: The GPU scheduling optimization method for a deep learning inference service system provided by the present invention includes the following steps:

[0054] S1. Initialize the scheduling optimization parameters, specifically including:

[0055] The system-defined global shared data includes the model variable array models[2][m], the model state index si, and the model state lock variable si_lock corresponding to the model state index si. A simulation queue and a request queue are set for each model. Among them, models[2][m] is used to store state information, and the stored state information includes the estimated value of the throughput requirement of each model and the actual throughput of the current cycle. m is the number of models. Each column of the models[2][m] array corresponds to the state information of one model. The initial values of the elements in models[2][m] and si are set to 0, and the value of si is limited to 0 or 1. To avoid read-write conflicts of the model state information during the prediction process and the scheduling process, the scheduling process only reads and writes the model states stored in each column of the models[2][m] array from the row specified by the si value, while the prediction process reads and writes the model states stored in each column of the models[2][m] array from the row specified by the si′ value, where si′ = (si + 1) % 2, and % is the modulo symbol. The number of parallel prediction processes is n, and the number of parallel scheduling processes is k. m, n, and k are all natural integers, and the value of m needs to satisfy being greater than or equal to 1. The specific values of n and k are set according to the hardware configuration of the system. Each prediction process or scheduling process is executed by a thread respectively. The request queue of each model is used to store the inference requests waiting for response of the model, and the simulation queue is used to store the inference requests involved in the prediction process described in step S2.

[0056] S2. Obtain the tasks to be processed by the deep learning inference service system at the current moment, and predict the throughput requirements of each model in the new round of cycle based on the system parameters of the deep learning inference service system, specifically including:

[0057] As Figure 3 shown in the overall flowchart of throughput requirement prediction: Among them, the model throughput refers to the number of successfully responded inference requests of the model. Each inference request has a deadline. Only when the client obtains the corresponding inference result before the deadline of the request, the request is considered to be successfully responded and counted into the throughput of the model.

[0058] The following steps are used to predict the throughput requirements of each model in the new round of cycle:

[0059] A. Evenly distribute all models to each prediction process: The number of models in the system is m, and the number of parallel prediction processes is n. m and n are both natural numbers, and the value of m satisfies being greater than or equal to 1. The specific value of n is set according to the hardware configuration of the system and is greater than or equal to 1. The number of models assigned to the first n - 1 prediction processes is ones, is the ceiling symbol, and the number of models allocated to the last prediction process is m % n, where % is the remainder operation;

[0060] B. The prediction thread estimates the throughput requirements of all models in the new cycle; for the current model, the prediction thread assumes that the model exclusively occupies the GPU computing resources, simulates and schedules the requests that have not been completed for the model, calculates the number of inference requests that can be successfully responded to within the duration of one cycle under the simulated scheduling situation, and takes the calculation result as the throughput requirement of the model in the new cycle;

[0061] C. All prediction processes end after completing the estimation task;

[0062] D. Obtain the model status lock variable si_lock, update the model status index si to si = si', publish the model status data in the new cycle for the scheduling process, and release the model status lock variable after the update ends;

[0063] E. End this round of prediction process;

[0064] As Figure 4 shown is the detailed flowchart of the prediction process for estimating the throughput requirements of a single model in the new cycle:

[0065] The detailed process of the prediction thread estimating the throughput requirements for a single model specifically includes:

[0066] (1) Copy all requests in the request queue of the current model to the simulation queue of the model;

[0067] (2) Clear the status data to be published for the current model: Assume the number of the current model is i, and the element corresponding to models[(si + 1) % 2][i] is the status data to be published for the model; Each model status data contains two member variables: sim_solo and goodput; Among them, models[(si + 1) % 2][i].sim_solo is the sim_solo member variable of the model numbered i, used to record the throughput requirement estimated for the current model, and models[(si + 1) % 2][i].goodput is the goodput member variable of the model numbered i, used to record the actual throughput of the current model; Assign the sim_solo and goodput member variables of each model status data to 0 to clear the status data to be published for the current model;

[0068] (3) Delete the requests in the simulation queue of the current model that cannot be completed within their own deadlines;

[0069] (4) Set the start and end variables to indicate the start time and end time of a new round of the cycle respectively; read the current system time and assign it to start; then assign the result of adding start and the cycle duration to end;

[0070] (5) Assume the system has N GPUs, set the simulation scheduling time sched_time for each GPU, and initialize sched_time to start;

[0071] (6) Determine whether the simulation queue is empty: if it is empty, execute step (13); otherwise, execute step (7);

[0072] (7) Obtain the minimum sched_time value among all GPUs and assign the result to the variable min_sched_time. min_sched_time is a temporary variable that records the minimum sched_time value among all GPUs; set the GPU with the minimum sched_time value as GPU i ;

[0073] (8) Determine whether min_sched_time is greater than or equal to end: if so, execute step (13); otherwise, execute step (9);

[0074] (9) Search for as many consecutive request blocks as possible from the simulation queue and meet the following requirements:

[0075] min_sched_time + inferTime(batch_size) < deadline

[0076] where the deadline for the first request in the request block is deadline, the number of requests in the block is batch_size; inferTime(batch_size) is the completion time for batch executing batch_size current model requests; the above formula is used to determine whether the request block can be batch executed on the GPU when the current model is exclusive to the GPU;

[0077] (10) Update the throughput of the current model in the simulation scheduling situation to sim_solo + batch_size; where sim_solo is a member variable of the model status data;

[0078] (11) Update the simulation scheduling time sched_time of GPU i in the simulation scheduling situation to min_sched_time + inferTime(batch_size);

[0079] (12) Delete all requests in the consecutive request block described in step (9) from the simulation queue, and jump to step (6);

[0080] (13) End the prediction process for the current model;

[0081] S3. Start a new round of cycle. Based on the throughput requirements of each model obtained in step S2 during the new round of cycle, adjust and allocate the throughput of each model within the cycle duration; specifically including the following steps:

[0082] As Figure 5 shown in the schematic diagram of the throughput adjustment process based on the feedback control strategy, the specific process of each scheduling thread performing a single throughput adjustment includes:

[0083] 1) Obtain the model status lock variable si_lock;

[0084] 2) Copy all elements in the row indicated by the model status index si in the variable array models[2][m] to the local one-dimensional array ms, and then release the model status lock variable si_lock;

[0085] 3) Traverse each element in the array ms and judge whether the actual throughput obtained by each model is lower than the standard value: If then the actual throughput obtained by this model is lower than the standard value; m i is the (i + 1)-th element in the array ms;

[0086] 4) If the actual throughput of the model is lower than the standard value, add these models to the M list with an initial value of empty; otherwise, add all models to the M list;

[0087] 5) Find as many consecutive request blocks as possible from the request queues of each model to meet the following requirements:

[0088] curTime + inferTime(batch_size) < deadline

[0089] where the arrival time of the first request in the request block is arrival_time, the deadline is deadline, the number of requests in the block is batch_size, the current system time is curTime, and inferTime(batch_size) is the time to execute batch_size requests of the current model in batch; deadline - inferTime(batch_size) is the latest scheduling time when all requests in the request block can be successfully responded; then set the time for the next scheduling process for the current GPU to curTime + inferTime(batch_size);

[0090] 6) Exit the scheduling process.

[0091] Place 1 GoogleNet and ResNet152 models on a single NVIDIA Tesla V100 GPU, and then apply a load of 325 inference requests per second to each model to test the effective throughput (Goodput) per second of each model supported by Clockwork and the method of the present invention.

[0092] As Figure 6 shown is a schematic diagram of the performance comparison between the present invention and the prior art. The experimental results are as Figure 6 shown. It can be observed from the experimental results that the present invention can significantly improve the Goodput of the GoogleNet model with little impact on the relatively large ResNet152 model, meeting the real-time requirements of more inference service requests.

[0093] S4. After the current cycle ends, repeat steps S2 - S3 until the deep learning inference service system terminates operation, completing the GPU scheduling optimization for the deep learning inference service system.

Claims

1. A GPU scheduling optimization method for a deep learning inference service system, comprising the following steps: S1. Initialize the scheduling optimization parameters; Specifically include: The globally shared data defined by the system includes a model variable array models[2][m], a model status index si, and a model status lock variable si_lock corresponding to the model status index si. A simulation queue and a request queue are set for each model. Among them, models[2][m] is used to store status information, and the stored status information includes the estimated value of the throughput requirement of each model and the actual throughput of the current cycle. m is the number of models, and each column of the models[2][m] array corresponds to the status information of one model. Set the initial values of each element in models[2][m] and si to 0, and the value range of si is limited to 0 or 1. To avoid read-write conflicts on the model status information during the prediction process and the scheduling process, the scheduling process only reads and writes the model status stored in each column of the models[2][m] array from the row specified by the si value, and the prediction process reads and writes the model status stored in each column of the models[2][m] array from the row specified by the si′ value, where si′ = (si + 1) % 2, and % is the modulo symbol. The number of parallel prediction processes is n, and the number of parallel scheduling processes is k. m, n, and k are all natural integers, and the value of m needs to satisfy being greater than or equal to 1. The specific values of n and k are set according to the hardware configuration of the system. The request queue of each model is used to store the inference requests waiting for response of the model, and the simulation queue is used to store the inference requests involved in the prediction process of step S2; S2. Obtain the tasks to be processed by the deep learning inference service system at the current moment, and predict the throughput requirements of each model in the new round of cycle based on the system parameters of the deep learning inference service system; Specifically, the following steps are adopted to predict the throughput requirements: A. Distribute all models evenly among each prediction process: The number of models in the system is m, and the number of prediction processes executed in parallel is n. Both m and n are natural numbers, m satisfies the condition of being greater than or equal to 1, and the specific value of n is set according to the hardware configuration of the system and is greater than or equal to 1. The number of models assigned to the first n - 1 prediction processes is pieces, is the ceiling symbol, and the number of models assigned to the last prediction process is m % n, where % is the remainder operation; B. Start the prediction process to estimate the throughput requirements of all models in the new round of cycle: For the current model, the prediction process assumes that the model monopolizes the GPU computing resources, simulates the scheduling of the requests that the model has not completed, calculates the number of inference requests that can be successfully responded within one cycle duration under the simulated scheduling situation, and takes the calculation result as the throughput requirement of the model in the new round of cycle; C. All prediction processes end after completing the estimation task; D. Obtain the model status lock variable si_lock, update the model status index si to si = si', publish the model status data in the new round of cycle for the scheduling process, and release the model status lock variable after the update ends; E. End this round of prediction process; S3. Start a new round of cycle, and based on the throughput requirements of each model obtained in step S2, adjust and allocate the throughput of each model within the cycle duration; S4. After the current cycle ends, repeat steps S2 - S3 until the deep learning inference service system terminates operation, and complete the GPU scheduling optimization for the deep learning inference service system.

2. The GPU scheduling optimization method for the deep learning inference service system according to claim 1, wherein Based on the system parameters of the deep learning inference service system described in step S2, predict the throughput requirements of each model in the to-be-processed task during a new round of cycles. Specifically, the following steps are used to predict the throughput requirements: The detailed process of predicting the throughput requirements for a single model during the prediction process specifically includes: (1) Copy all requests in the request queue of the current model to the simulation queue of this model; (2) Clear the status data to be published for the current model: Assume the number of the current model is i, and the element corresponding to models[(si + 1) % 2][i] is the status data to be published for this model; Each model status data contains two member variables: sim_solo and goodput; Among them, models[(si + 1) % 2][i].sim_solo is the sim_solo member variable of the model numbered i, which is used to record the throughput requirements estimated for the current model, and models[(si + 1) % 2][i].goodput is the goodput member variable of the model numbered i, which is used to record the actual throughput of the current model; Assign the sim_solo and goodput member variables of each model status data to 0 to clear the status data to be published for the current model; (3) Delete requests in the simulation queue of the current model that cannot be completed within their own deadlines; (4) Set start and end variables, which are used to indicate the start time and end time of a new round of cycles respectively; Read the current system time and assign it to start; Then assign the result of adding start and the cycle duration to end; (5) Assume the system has N GPUs, set the simulation scheduling time sched_time for each GPU, and initialize sched_time to start; (6) Determine whether the simulation queue is empty: If it is empty, execute step (13); Otherwise, execute step (7); (7) Obtain the minimum sched_time value among all GPUs, and assign the result to the variable min_sched_time. min_sched_time is a temporary variable that records the minimum sched_time value among all GPUs; Set the GPU with the minimum sched_time value as GPU i ; (8) Determine whether min_sched_time is greater than or equal to end: If so, execute step (13); Otherwise, execute step (9); (9) Find as many consecutive request blocks as possible in the simulation queue and meet the following requirements: min_sched_time + inferTime(batch_size) < deadline where the deadline of the first request in the request block is deadline, the number of requests in the block is batch_size; inferTime(batch_size) is the completion time for batch executing batch_size requests of the current model; The above formula is used to determine whether the request block can be batch executed on the GPU when the current model occupies the GPU alone; (10) Update the throughput of the current model in the simulation scheduling case to sim_solo + batch_size; where sim_solo is a member variable of the model status data; (11) Update the simulated scheduling time sched_time of the GPU under the simulated scheduling scenario to min_sched_time + inferTime(batch_size); i The simulated scheduling time sched_time of (12) Delete all requests in the consecutive request block described in step (9) from the simulation queue, and jump to step (6); (13)End the prediction process for the current model.

3. The GPU scheduling optimization method for a deep learning inference service system according to claim 2, wherein Based on the throughput requirements of each model obtained in step S2 in the new round of cycle, adjust and allocate the throughput of each model in the new round of cycle, which specifically includes the following steps: 1) Obtain the model status lock variable si_lock; 2) Copy all elements in the row indicated by the model status index si in the variable array models[2][m] to the local one-dimensional array ms, and then release the model status lock variable si_lock; 3) Traverse each element in the array ms and determine whether the actual throughput obtained by each model is lower than the standard value: If then the actual throughput obtained by this model is lower than the standard value; m i is the (i + 1)-th element in the array ms; 4) If the actual throughput of the model is lower than the standard value, add these models to the M list with an initial value of empty; otherwise, add all models to the M list; 5) Find as many consecutive request blocks as possible from the request queues of each model to meet the following requirements: curTime+inferTime(batch_size)<deadline where the arrival time of the first request in the request block is arrival_time, the deadline is deadline, the number of requests in the block is batch_size, the current system time is curTime, and inferTime(batch_size) is the time to execute batch_size current model requests in batch; deadline-inferTime(batch_size) is the latest scheduling time when all requests in the request block can be successfully responded; then set the time for the next scheduling process for the current GPU to curTime+inferTime(batch_size); 6) Exit the scheduling process.

Images