Multi-tenant task scheduling method and system

Abstract

The application provides a multi-tenant task scheduling method and system. The method comprises: generating a candidate layer set meeting QoS requirements for each layer of a deep neural network according to a user task request; setting an identifier for each candidate layer in the candidate layer set and reconstructing a deep neural network model; calculating co-location overhead of different candidate layer combinations and generating a co-location overhead set in the case of executing the deep neural network model in a GPU, and selecting at least one candidate layer combination with the best co-location performance as an optimal candidate implementation; selecting an optimal candidate layer in the candidate layer set as the optimal candidate implementation in the case of not executing the deep neural network model in the GPU; writing the optimal candidate implementation as an identifier list into the user task request and submitting the user task request to the GPU; and selecting a corresponding candidate layer implementation according to the identifier list by a GPU runtime. The method can guarantee QoS of each job and improve the throughput of a multi-tenant system.

Description

Technical Field

[0001] This invention relates to the field of GPU technology, and in particular to a multi-tenant task scheduling method and system. Background Technology

[0002] Deep neural networks place high demands on the computing and storage units of hardware computing platforms. However, a single deep neural network typically cannot simultaneously utilize the computing power and memory bandwidth of a GPU. Some computationally intensive tasks can make good use of computing units but use less memory bandwidth; while memory-intensive applications occupy memory bandwidth while leaving some computing units idle, resulting in wasted computing power. Furthermore, the request volume for model services fluctuates over time. When the request volume is low at a certain moment, resources become idle, while when the request volume surges, multiple requests executing on the same accelerator can cause resource contention, thus affecting the Quality of Service (QoS).

[0003] To improve hardware resource utilization efficiency and inference service throughput, a typical approach is to deploy multiple deep learning model services on the same hardware. To enable accelerators to execute multiple inference tasks from different model services simultaneously, common accelerator reuse mechanisms are implemented differently at different hardware and software layers. A typical hardware-level multi-model execution mechanism is single-root I / O virtualization (SR-IOV), which divides the entire accelerator's computing and storage resources into multiple smaller accelerators. Different model services can be deployed in different smaller accelerators. The advantage of this virtualization-based reuse method is low interference between models; the disadvantage is that resources are not elastically adaptable, making it difficult to adapt to fluctuating inference request flows. Reallocating resources requires restarting the service, which takes a considerable amount of time. Runtime-level reuse mechanisms include Nvidia's Multi-Process Service (MPS) mechanism, which isolates only the computing resources between multiple models, not the storage resources. In addition, the runtime layer also includes multi-stream-based multi-job execution mechanisms, which cannot isolate either computing or storage resources. The advantage of software-based accelerator reuse is that resources are more elastic, but isolation is weak and resources compete between services, thus failing to guarantee QoS. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a multi-tenant task scheduling method and system that improves the throughput of the multi-tenant system while ensuring the QoS of each job.

[0005] To achieve the above objectives, the present invention provides a multi-tenant task scheduling method, comprising:

[0006] Obtain user task requests, and based on the user task requests, generate a set of candidate layers that meet QoS requirements for each layer of the deep neural network;

[0007] Assign an identifier to each candidate layer in the candidate layer set, and reconstruct the deep neural network model based on the candidate layer set;

[0008] Determine whether the deep neural network model is being executed on the GPU.

[0009] When executing the aforementioned deep neural network model

[0010] Calculate the common cost of different candidate layer combinations and generate a common cost set.

[0011] Select at least one set of candidate layer combinations with the best co-location performance as the optimal candidate implementation;

[0012] Without executing the deep neural network model, the optimal candidate layer in the candidate layer set is selected as the optimal candidate implementation;

[0013] The optimal candidate implementation is written as an identifier list into the user task request and submitted to the GPU;

[0014] The GPU runtime selects the corresponding candidate layer implementation based on the identifier list.

[0015] Optionally, generating a set of candidate layers that meet QoS requirements for each layer of the deep neural network includes:

[0016] The optimal number of thread blocks in the current layer of the deep neural network was obtained by tuning when all computing units were used.

[0017] Adjust the number of computing units so that the thread blocks implemented in the optimal layer are evenly distributed among different computing units, and test the performance when using different numbers of computing units respectively;

[0018] By quantifying the computation time of each layer, the proportion of execution time occupied by each layer in the execution flow of the deep neural network is calculated, and combined with the QoS of the entire model, the upper bound of the QoS of each layer is obtained.

[0019] The number of computing units that do not meet the QoS requirements is excluded by using the QoS upper bound, and the number of computing units that meet the QoS requirements is obtained.

[0020] The number of computing units that meet QoS requirements is re-optimized to generate the layer implementation, forming the candidate layer set for each layer.

[0021] Optionally, reconstructing the deep neural network model based on the candidate layer set includes:

[0022] Traverse the directed acyclic graph of the deep neural network, add the candidate layer set to the corresponding layer of the model, and reconstruct the deep neural network model.

[0023] Optionally, determining whether the deep neural network model is being executed in the GPU includes:

[0024] By creating memory of bitmap type, the runtime state of the GPU can be monitored;

[0025] Each index in the bitmap represents the topological sorting index of the current layer in the deep neural network, and each index stores the running progress of the current layer.

[0026] Optionally, calculating the co-location cost of different candidate layer combinations and generating a co-location cost set includes:

[0027] Determine the time difference between the time delay of the isolated operation of the previous candidate layer and the time delay of the co-operation of the current candidate layer in the deep neural network, as well as the overlap time between the previous candidate layer and the current candidate layer;

[0028] Based on the ratio between the time difference and the overlap, the colocation cost per unit time is calculated.

[0029] Generate a set of common overheads for different candidate layer combinations.

[0030] Optionally, selecting a set of candidate layer combinations with the best co-location performance as the optimal candidate implementation includes:

[0031] When a current user task request arrives, obtain GPU runtime status information, including the number of request streams currently being executed, the execution progress of each request stream, and the candidate implementation types of the remaining execution operators in each request stream;

[0032] Based on the GPU runtime state information, select the set of common overhead for the current user task request;

[0033] Select at least one set of candidate layer combinations that satisfy the preset common overhead from the set of common overheads as the optimal candidate implementation.

[0034] Optionally, at least one set of candidate layer combinations that satisfy the preset co-location cost is selected from the set of co-location costs as the optimal candidate implementation, including:

[0035] For each candidate layer, calculate the execution time for all possible co-location combinations;

[0036] The smaller value of the product of the execution time corresponding to each co-location combination and the co-location cost per unit time is taken as the co-location execution time, and the co-location execution time is used as the preset co-location cost.

[0037] Optionally, selecting the optimal candidate layer from the candidate layer set as the optimal candidate implementation includes:

[0038] Traverse the candidate layer set, calculate the total execution time of all candidate layer combinations, and select the candidate layer with the shortest execution time as the optimal candidate implementation.

[0039] In another aspect, the present invention provides a multi-tenant task scheduling system, employing the above-described multi-tenant task scheduling method, wherein the system comprises at least:

[0040] A candidate layer generator is used to obtain user task requests and, based on the user task requests, generate a set of candidate layers that meet QoS requirements for each layer of the deep neural network.

[0041] A neural network model reconstructor is used to assign an identifier to each candidate layer in the candidate layer set and reconstruct a deep neural network model based on the candidate layer set.

[0042] A resource monitor is used to determine whether the deep neural network model is being executed on the GPU.

[0043] A job colocation estimator is used to calculate the colocation cost of different candidate layer combinations and generate a colocation cost set when the deep neural network model is executed.

[0044] The layer scheduler, linking the resource monitor and the job colocation estimator, is used for:

[0045] When executing the deep neural network model, at least one set of candidate layer combinations with the best co-location performance is selected as the optimal candidate implementation;

[0046] Without executing the deep neural network model, the optimal candidate layer from the candidate layer set is selected as the optimal candidate implementation; and

[0047] The optimal candidate implementation is written as an identifier list into the user task request and submitted to the GPU;

[0048] The GPU runtime selects the corresponding candidate layer implementation based on the identifier list.

[0049] Optionally, the layer scheduler is further configured to:

[0050] When a current user task request arrives, obtain GPU runtime status information, including the number of request streams currently being executed, the execution progress of each request stream, and the candidate implementation types of the remaining execution operators in each request stream;

[0051] Based on the GPU runtime state information, select the set of common overhead for the current user task request;

[0052] Traverse the candidate layer set, calculate the total execution time of all candidate layer combinations, and select the candidate layer with the shortest execution time as the optimal candidate implementation;

[0053] For each candidate layer, calculate the execution time for all possible co-location combinations;

[0054] The smaller value of the product of the execution time corresponding to each co-location combination and the co-location cost per unit time is taken as the co-location execution time, and the co-location execution time is used as the preset co-location cost.

[0055] Select at least one set of candidate layer combinations that satisfy the preset common overhead as the optimal candidate implementation.

[0056] As can be seen from the above solutions, the advantages of the present invention are:

[0057] The multi-tenant task scheduling method and system provided by this invention obtains user task requests and generates a set of candidate layers that meet QoS requirements for each layer of a deep neural network based on the user task requests; assigns an identifier to each candidate layer in the candidate layer set; reconstructs a deep neural network model based on the candidate layer set; determines whether the deep neural network model is being executed on the GPU; if the deep neural network model is being executed, calculates the co-location cost of different candidate layer combinations, generates a co-location cost set, and selects at least one set of candidate layer combinations with the best co-location performance as the optimal candidate implementation; if the deep neural network model is not being executed, selects the best candidate layer in the candidate layer set as the optimal candidate implementation; writes the optimal candidate implementation as an identifier list into the user task request and submits it to the GPU; the GPU selects the corresponding candidate layer implementation based on the identifier list during runtime. This method analyzes deep learning co-location jobs, tests the performance of each deep neural network layer under different numbers of computing units, compiles and generates a set of candidate implementations under computing units that meet QoS, and the GPU dynamically calculates the co-location cost of different layer combinations based on runtime resource usage and selects the optimal co-location implementation, thereby improving the throughput of the multi-tenant system while ensuring the QoS of each job. Attached Figure Description

[0058] Figure 1 This is a flowchart illustrating the multi-tenant task scheduling method provided in an embodiment of the present invention.

[0059] Figure 2 Provided for embodiments of the present invention Figure 1 A detailed flowchart of step S1;

[0060] Figure 3 This is a framework diagram of a multi-tenant task scheduling system provided in an embodiment of the present invention;

[0061] in:

[0062] 200-Multi-tenant Task Scheduling System;

[0063] 201-Candidate Layer Generator;

[0064] 202 - Neural Network Model Reconstructor;

[0065] 203 - Resource Monitor;

[0066] 204 - Job co-location estimator;

[0067] 205-layer scheduler;

[0068] 206-GPU. Detailed Implementation

[0069] To make the above features and effects of the present invention clearer and easier to understand, specific embodiments are described below, and detailed descriptions are provided in conjunction with the accompanying drawings.

[0070] This invention proposes a code generation and adaptive multi-tenant task scheduling method, which accelerates the low-interference and high-throughput of multi-tenant deep learning services.

[0071] For details, please refer to Figure 1 As shown, Figure 1 A flowchart illustrating the multi-tenant task scheduling method is shown.

[0072] A multi-tenant task scheduling method includes:

[0073] S1. Obtain user task requests, and based on the user task requests, generate a set of candidate layers that meet QoS requirements for each layer of the deep neural network.

[0074] In specific implementations, such as Figure 2 As shown, generating a set of candidate layers that meet QoS requirements for each layer of the deep neural network includes:

[0075] S11. Test the number of thread blocks in the optimal layer implementation obtained by tuning when the current layer uses all computing units in the deep neural network;

[0076] S12. Adjust the number of computing units so that the thread blocks implemented in the optimal layer are evenly distributed among different computing units, and test the performance when using different numbers of computing units respectively.

[0077] S13. By quantifying the computation time of each layer, calculate the proportion of execution time occupied by each layer in the execution flow of the deep neural network, and combine it with the QoS of the entire model to obtain the upper bound of QoS for each layer.

[0078] S14. Use the QoS upper bound to exclude the number of computing units that do not meet the QoS requirements, and obtain the setting of the number of computing units that meet the QoS requirements.

[0079] S15. Re-optimize the number of computing units that meet the QoS requirements, generate the layer implementation, and form the candidate layer set for each layer.

[0080] S2. Assign an identifier to each candidate layer in the candidate layer set, and reconstruct the deep neural network model based on the candidate layer set.

[0081] In a specific implementation, reconstructing the deep neural network model based on the candidate layer set includes:

[0082] Traverse the directed acyclic graph of the deep neural network, add the candidate layer set to the corresponding layer of the model, and reconstruct the deep neural network model.

[0083] S3. Determine whether the deep neural network model is being executed in the GPU.

[0084] In a specific implementation, determining whether the deep neural network model is being executed in the GPU includes:

[0085] By creating memory of bitmap type, the runtime state of the GPU can be monitored;

[0086] Each index in the bitmap represents the topological sorting index of the current layer in the deep neural network, and each index stores the running progress of the current layer.

[0087] S41. When executing the deep neural network model...

[0088] Calculate the common overhead for different candidate layer combinations, and generate a set of common overheads, including:

[0089] Determine the time difference between the time delay of the isolated operation of the previous candidate layer and the time delay of the co-operation of the current candidate layer in the deep neural network, as well as the overlap time between the previous candidate layer and the current candidate layer;

[0090] Based on the ratio between the time difference and the overlap, the colocation cost per unit time is calculated.

[0091] Generate a set of common overheads for different candidate layer combinations.

[0092] Then, at least one set of candidate layer combinations with the best co-location performance is selected as the optimal candidate implementation, including:

[0093] When a current user task request arrives, obtain GPU runtime status information, including the number of request streams currently being executed, the execution progress of each request stream, and the candidate implementation types of the remaining execution operators in each request stream;

[0094] Based on the GPU runtime state information, select the set of common overhead for the current user task request;

[0095] Select at least one set of candidate layer combinations that satisfy the preset common overhead from the set of common overheads as the optimal candidate implementation.

[0096] In a specific implementation, at least one set of candidate layer combinations that satisfy the preset common overhead is selected from the set of common overhead as the optimal candidate implementation, including:

[0097] For each candidate layer, calculate the execution time for all possible co-location combinations;

[0098] The smaller value of the product of the execution time corresponding to each co-location combination and the co-location cost per unit time is taken as the co-location execution time, and the co-location execution time is used as the preset co-location cost.

[0099] S42. Without executing the deep neural network model, select the optimal candidate layer from the candidate layer set as the optimal candidate implementation.

[0100] In a specific implementation, selecting the optimal candidate layer from the candidate layer set as the optimal candidate implementation includes:

[0101] Traverse the candidate layer set, calculate the total execution time of all candidate layer combinations, and select the candidate layer with the shortest execution time as the optimal candidate implementation.

[0102] S5. The optimal candidate implementation is written into the user task request as an identifier list and submitted to the GPU;

[0103] The GPU runtime selects the corresponding candidate layer implementation based on the identifier list.

[0104] In summary, the multi-tenant task scheduling method provided in this embodiment obtains user task requests and generates a set of candidate layers that meet QoS requirements for each layer of a deep neural network based on these requests. Each candidate layer in the set is assigned an identifier, and the deep neural network model is reconstructed based on the set. The method then determines whether the deep neural network model is being executed on the GPU. If the model is being executed, the co-location overhead of different candidate layer combinations is calculated, a co-location overhead set is generated, and at least one set of candidate layer combinations with the best co-location performance is selected as the optimal candidate implementation. If the deep neural network model is not being executed, the optimal candidate layer in the set is selected as the optimal candidate implementation. The optimal candidate implementation is written as an identifier list into the user task request and submitted to the GPU. The GPU selects the corresponding candidate layer implementation based on the identifier list during runtime. This method analyzes deep learning co-location jobs, tests the performance of each deep neural network layer under different numbers of computing units, compiles and generates a set of candidate implementations under QoS-compliant computing units, and the GPU dynamically calculates the co-location overhead of different layer combinations based on runtime resource usage and selects the optimal co-location implementation. This improves the throughput of the multi-tenant system while ensuring the QoS of each job.

[0105] The above embodiments of the present invention can be applied to terminal devices that implement multi-tenant task scheduling methods. These terminal devices may include personal terminals and host computer terminals, etc., and the embodiments of the present invention do not impose any limitations on them. The terminal can support operating systems such as Windows, Android, iOS, and Windows Phone.

[0106] Reference Figure 3 , Figure 3 A multi-tenant task scheduling system 200 is shown. The multi-tenant task scheduling method can be applied to personal terminals and host computer terminal devices, and can achieve the following through, for example Figure 1 The multi-tenant task scheduling method shown in this application embodiment and the multi-tenant task scheduling system provided can implement each process of the above-described multi-tenant task scheduling method.

[0107] A multi-tenant task scheduling system 200, comprising at least:

[0108] The candidate layer generator 201 is used to obtain user task requests and, based on the user task requests, generate a set of candidate layers that meet QoS requirements for each layer of the deep neural network.

[0109] In this embodiment, the candidate layer generator 201 generates a set of candidate layer implementations that meet QoS requirements for each layer of the deep neural network. The difference between different layer implementations lies in the number of thread blocks. Layers in the neural network are divided into thread blocks of equal size. Each thread block executes in only one computational unit during runtime scheduling. Fewer thread blocks result in more threads per block, meaning stronger locality and weaker parallelism in the kernel function; more thread blocks result in fewer threads per block, meaning poorer parallelism in the kernel function. By measuring parallelism and locality based on the computational and memory access characteristics of different layers, and considering the number of available computational units, selecting an appropriate number of thread blocks can effectively accelerate the computation process of the neural network. The candidate layer generator optimizes the layers offline using the TVM's AutoScheduler code generator and generates a set of candidate layer implementations that meet QoS requirements.

[0110] Specifically, firstly, the optimal number of thread blocks for the current layer implementation in the deep neural network is obtained by testing and optimizing the implementation using all computing units. Simultaneously, the number of computing units is adjusted to evenly distribute the thread blocks of the optimal layer implementation across different computing units, and performance is tested with different numbers of computing units. Then, by quantifying the computation time of each layer, the proportion of execution time for each layer in the deep neural network execution flow is calculated. Combined with the overall QoS of the model, a QoS upper bound for each layer is obtained. Simultaneously, the number of computing units that do not meet the QoS requirements is excluded using the QoS upper bound, and the number of computing units that meet the QoS requirements is obtained. Finally, the implementation is re-optimized with the number of computing units that meet the QoS requirements, generating layer implementations that form the candidate layer set for each layer.

[0111] The neural network model reconstructor 202, linked to the candidate layer generator 201, is used to set an identifier for each candidate layer in the candidate layer set and reconstruct the deep neural network model based on the candidate layer set.

[0112] In this embodiment, the deep neural network model is reconstructed by rewriting the directed acyclic graph (DAG) of the deep neural network model in the request, traversing the DAG, and adding the candidate layer set to the corresponding layer of the model. Specifically, the neural network request needs to specify the kernel function and the corresponding number of thread blocks for each layer, forming a defined task execution flow. The model reconstructor uses an integration layer to map the candidate set and identifiers to the task request, mapping the execution flow one-to-one with the list of identifiers selected by the scheduler. This allows the scheduler to dynamically adjust the task execution flow, sharing overhead for subsequent computation tasks, thereby providing space for optimizing system throughput.

[0113] Resource monitor 203 is used to determine whether the deep neural network model is being executed in the GPU.

[0114] In its implementation, the resource monitor 203 monitors the execution progress of each layer in the neural network and calculates the co-location cost to select the optimal co-location combination. This requires not only the model information for the current request but also information about the tasks co-located on the GPU. The resource monitor creates bitmaps in both GPU and CPU memory. Each index in the bitmap represents a layer's topological sort index in the network, and the corresponding bitmap value represents the current layer's execution progress. Code to update the bitmaps is inserted into the kernel function using code generation. Whenever a layer iterates a certain number of times, the thread in the kernel atomically updates the corresponding bitmap value in GPU memory and periodically performs GPU-to-CPU memory synchronization. When a requested model finishes execution, all values ​​in the bitmap become invalid.

[0115] The job colocation estimator 204 is used to calculate the colocation cost of different candidate layer combinations and generate a colocation cost set when the deep neural network model is executed.

[0116] In its implementation, the job co-location estimator 204 is used to calculate the co-location overhead for different layer combinations. Quantifying job co-location overhead is crucial for measuring the cost-effectiveness of improved system throughput versus reduced job latency. If the model performance overhead in a multi-tenant scenario cannot be predicted, the QoS requirements for real-time tasks in real-world systems cannot be met, and improved system throughput becomes impossible. The estimator offline tests the co-location overhead of all model layer combinations. Simultaneously, the job co-location estimator calculates the time difference between the latency of the previous candidate layer running in isolation and the latency of the current candidate layer running in co-location within the deep neural network, as well as the overlap time between the previous and current candidate layers. Based on the ratio between the time difference and the overlap, it derives the co-location overhead per unit time, thereby generating a set of co-location overheads for different candidate layer combinations.

[0117] Layer scheduler 205, linked to the resource monitor, job co-location estimator, neural network model refactorer 202, and GPU 206, respectively calls the runtime resource monitor and job co-location estimator to obtain the current computation progress and co-location cost set of the latest layer, and calculates an identifier list using an adaptive layer scheduling algorithm. The layer scheduler then writes the identifier list into the data field of the user task request, enabling the GPU to select the corresponding layer for implementation based on the identifier list. In other words, the execution flow of the neural network corresponds one-to-one with the identifier list determined by the layer scheduling algorithm, improving system throughput while satisfying QoS.

[0118] Specifically, when executing the deep neural network model, the layer scheduler selects at least one set of candidate layer combinations with the best co-location performance as the optimal candidate implementation; when not executing the deep neural network model, the layer scheduler selects the best candidate layer from the candidate layer set as the optimal candidate implementation; then, the layer scheduler uses the optimal candidate implementation as an identifier list, writes it into the user task request and submits it to the GPU, so that the GPU selects the corresponding candidate layer implementation according to the identifier list during runtime.

[0119] In a specific implementation, when a current user task request arrives, the layer scheduler 205 connects to the resource monitor to obtain GPU runtime status information monitored by the resource monitor. This GPU runtime status information specifically includes the number of currently executing request streams, the execution progress of each request stream, and the candidate implementation types of the remaining execution operators in each request stream. Simultaneously, the layer scheduler 205 connects to the job co-location estimator 204 and, based on the obtained GPU runtime status information, selects a set of co-location costs for the current user task request. Then, the layer scheduler iterates through the candidate layer set. After completing the iteration, it calculates the total execution time of all candidate layer combinations and selects the candidate layer with the shortest execution time as the optimal candidate implementation. While traversing the candidate layer set is not yet complete, for each candidate layer, the execution time of all possible co-location combinations is calculated. Then, the smaller value of the product of the execution time corresponding to each co-location combination and the co-location overhead per unit time is taken as the co-location execution time. Based on the co-location execution time, i.e., the co-location execution time is used as the preset co-location overhead, at least one set of candidate layer combinations that meet the co-location execution time is selected as the optimal candidate implementation from the co-location overhead set. Finally, when the optimal candidate implementation is determined, the optimal candidate implementation is used as an identifier list by the layer scheduler, written into the user task request and submitted to the GPU, so that the GPU selects the corresponding candidate layer implementation according to the identifier list during runtime. At the same time, if the co-location model layer execution is completed, the index of the co-location model is further updated; if the co-location model layer execution is not completed, the layer index is retained and re-indexed by calculating the computation progress of the co-location model layer. This layer scheduler completes the selection of layer candidate implementations on the host CPU side by simulating operator co-location execution and uses multi-threaded parallelism to accelerate the entire computation process and avoid excessive scheduling time.

[0120] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be applied, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0121] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A multi-tenant task scheduling method, characterized in that, include: Obtain user task requests, and based on the user task requests, generate a set of candidate layers that meet QoS requirements for each layer of the deep neural network; Assign an identifier to each candidate layer in the candidate layer set, and reconstruct the deep neural network model based on the candidate layer set; Determine whether the deep neural network model is being executed in the GPU. If the deep neural network model is being executed, calculate the colocation cost of different candidate layer combinations, generate a colocation cost set, and select at least one candidate layer combination with the best colocation performance as the optimal candidate layer implementation. Without executing the deep neural network model, the optimal candidate layer in the candidate layer set is selected as the optimal candidate layer for implementation; The optimal candidate layer implementation is written as an identifier list into the user task request and submitted to the GPU; The GPU runtime selects the corresponding candidate layer implementation based on the identifier list; The step of generating a set of candidate layers that meet QoS requirements for each layer of the deep neural network includes: The number of thread blocks in the optimal candidate layer implementation obtained by testing the current layer of the deep neural network when all computing units are used is tested; Adjust the number of computing units so that the thread blocks implemented by the optimal candidate layer are evenly distributed among different computing units, and test the performance when using different numbers of computing units respectively; By quantifying the computation time of each layer, the proportion of execution time occupied by each layer in the execution flow of the deep neural network is calculated, and combined with the QoS of the entire model, the upper bound of the QoS of each layer is obtained. The number of computing units that do not meet the QoS requirements is excluded by using the QoS upper bound, and the number of computing units that meet the QoS requirements is obtained. The number of computing units that meet QoS requirements is re-optimized to generate candidate layer implementations, forming the candidate layer set for each layer.

2. The method according to claim 1, characterized in that, The process of reconstructing the deep neural network model based on the candidate layer set includes: Traverse the directed acyclic graph of the deep neural network, add the candidate layer set to the corresponding layer of the model, and reconstruct the deep neural network model.

3. The method according to claim 1, characterized in that, The determination of whether the deep neural network model is being executed in the GPU includes: By creating memory of bitmap type, the runtime state of the GPU can be monitored; Each index in the bitmap represents the topological sorting index of the current layer in the deep neural network, and each index stores the running progress of the current layer.

4. The method according to claim 3, characterized in that, The calculation of the common overhead for different candidate layer combinations, generating a common overhead set, includes: Determine the time difference between the time delay of the isolated operation of the previous candidate layer and the time delay of the co-operation of the current candidate layer in the deep neural network, as well as the overlap time between the previous candidate layer and the current candidate layer; Based on the ratio of the time difference to the overlap time, the colocation cost per unit time is calculated. Generate a set of common overheads for different candidate layer combinations.

5. The method according to claim 4, characterized in that, The selection of a set of candidate layer combinations with the best co-location performance as the optimal candidate layer implementation includes: When the current user task request arrives, obtain GPU runtime state information, including the number of request streams being executed, the execution progress of each request stream, and the candidate layer implementation type of the remaining execution operators in each request stream; Based on the GPU runtime state information, select the set of common overhead for the current user task request; Select at least one candidate layer combination that satisfies the preset common overhead from the set of common overheads as the optimal candidate layer implementation.

6. The method according to claim 5, characterized in that, Selecting at least one candidate layer combination that satisfies the preset common overhead from the set of common overheads as the optimal candidate layer implementation includes: For each candidate layer, calculate the execution time for all possible co-location combinations; The smaller value of the product of the execution time corresponding to each co-location combination and the co-location cost per unit time is taken as the co-location execution time, and the co-location execution time is used as the preset co-location cost.

7. The method according to claim 5, characterized in that, Selecting the optimal candidate layer from the candidate layer set as the optimal candidate layer implementation includes: Traverse the candidate layer set, calculate the total execution time of all candidate layer combinations, and select the candidate layer with the shortest execution time as the optimal candidate layer implementation.

8. A multi-tenant task scheduling system, characterized in that, The system employing the multi-tenant task scheduling method according to any one of claims 1-7, comprises at least: A candidate layer generator is used to obtain user task requests and, based on the user task requests, generate a set of candidate layers that meet QoS requirements for each layer of the deep neural network. A neural network model reconstructor is used to assign an identifier to each candidate layer in the candidate layer set and reconstruct a deep neural network model based on the candidate layer set. A resource monitor is used to determine whether the deep neural network model is being executed on the GPU. A job colocation estimator is used to calculate the colocation cost of different candidate layer combinations and generate a colocation cost set when the deep neural network model is executed. The layer scheduler, linking the resource monitor and the job colocation estimator, is used for: When executing the deep neural network model, at least one set of candidate layer combinations with the best co-location performance is selected as the optimal candidate layer implementation; Without executing the deep neural network model, the optimal candidate layer from the candidate layer set is selected as the optimal candidate layer for implementation; and The optimal candidate layer implementation is written as an identifier list into the user task request and submitted to the GPU; The GPU runtime selects the corresponding candidate layer implementation based on the identifier list; The step of generating a set of candidate layers that meet QoS requirements for each layer of the deep neural network includes: The number of thread blocks in the optimal candidate layer implementation obtained by testing the current layer of the deep neural network when all computing units are used is tested; Adjust the number of computing units so that the thread blocks implemented by the optimal candidate layer are evenly distributed among different computing units, and test the performance when using different numbers of computing units respectively; By quantifying the computation time of each layer, the proportion of execution time occupied by each layer in the execution flow of the deep neural network is calculated, and combined with the QoS of the entire model, the upper bound of the QoS of each layer is obtained. The number of computing units that do not meet the QoS requirements is excluded by using the QoS upper bound, and the number of computing units that meet the QoS requirements is obtained. The number of computing units that meet QoS requirements is re-optimized to generate candidate layer implementations, forming the candidate layer set for each layer.

9. The system according to claim 8, characterized in that, The layer scheduler is also used for: When the current user task request arrives, obtain GPU runtime state information, including the number of request streams being executed, the execution progress of each request stream, and the candidate layer implementation type of the remaining execution operators in each request stream; Based on the GPU runtime state information, select the set of common overhead for the current user task request; Traverse the candidate layer set, calculate the total execution time of all candidate layer combinations, and select the candidate layer with the shortest execution time as the optimal candidate layer implementation; For each candidate layer, calculate the execution time for all possible co-location combinations; The smaller value of the product of the execution time corresponding to each co-location combination and the co-location cost per unit time is taken as the co-location execution time, and the co-location execution time is used as the preset co-location cost. Select at least one set of candidate layer combinations that satisfy the preset common overhead as the optimal candidate layer implementation.

Images