GPU (Graphics Processing Unit) sharing scheduling method and device supporting heterogeneous operation scale

Abstract

The invention discloses a GPU sharing scheduling method and device supporting heterogeneous job scale, and the method comprises the steps: carrying out the interference portrait analysis of a newly submitted job, obtaining the interference feature representation of the job, and adding the interference feature representation into a job queue; collecting cluster operation information and job information; determining a job scheduling interval; on the basis of the job runtime information, the job priority is updated, and the jobs are sorted through a maximum sharing efficiency priority method; traversing jobs in the job queue, and determining jobs to be run and jobs to be preempted; calling an optimal packaging strategy and job co-located interference prediction model to determine a placement position of a to-be-operated job; placing jobs and updating a cluster state; and monitoring a new job submission condition and updating a scheduling interval, if a new job is submitted, skipping to carry out interference portrait analysis on the newly submitted job, otherwise, skipping to the step of determining the job scheduling interval. According to the method, the job completion time and the queuing time can be effectively reduced, and the resource scheduling efficiency of the GPU cluster is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of GPU scheduling, and in particular relates to a GPU sharing scheduling method and device supporting heterogeneous operation scales. Background Art

[0002] In recent years, with the rapid development of deep learning (DL) technology, it has been widely used in many fields such as image classification and recommendation systems. In order to further improve the capabilities of deep learning models, many information technology companies and research institutions have invested a lot of resources to build a wide range of deep learning infrastructure. In particular, the rise of large-scale language models (LLMs) has greatly promoted the demand for GPU computing power, which has also prompted cluster schedulers to play a key role in optimizing resource allocation and coordinating various deep learning tasks.

[0003] According to public cluster data records, there is a clear resource contention between single-GPU and multi-GPU workloads. Although the workload covers single-GPU and multi-GPU tasks, most training jobs tend to be executed using a single GPU. On the contrary, although the number of multi-GPU jobs is smaller, they usually take longer to complete. The current scheduling mechanism often gives priority to multi-GPU tasks, which often results in limited resource acquisition for single-GPU jobs, resulting in an imbalance in resource allocation.

[0004] One possible mitigation solution is to enable multiple deep learning jobs to run together on the same GPU. However, current methods are unable to cope with resource competition between multi-GPU and single-GPU jobs, especially in terms of how to effectively share resources between tasks with different GPU requirements. In addition, existing methods rely too much on experience-based packaging strategies, which are inadequate when faced with complex workload interference patterns. Therefore, scheduling multi-GPU and single-GPU jobs simultaneously in an environment that supports GPU sharing is a challenging task, and current schedulers perform poorly in this regard, resulting in suboptimal performance and efficiency of the entire system. Summary of the invention

[0005] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, and to provide a GPU sharing scheduling method and device that supports heterogeneous job scales, and to solve the problems caused by the co-location of multi-GPU and single-GPU workloads by analyzing task behaviors in various interference environments.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a GPU sharing scheduling method supporting heterogeneous job scales, comprising the following steps:

[0008] Perform interference profiling analysis on newly submitted jobs, obtain interference feature representations of the jobs, and add them to the job queue;

[0009] Collect cluster runtime information and job information, including the number of available GPUs in the cluster, the total number of GPUs requested by the job, the job queue, and the number of job preemption times;

[0010] Determine the job scheduling interval based on the number of available GPUs in the cluster, the total number of GPUs requested by the job, and the job queue information;

[0011] Based on the job runtime information, the job priority is updated and the jobs are sorted by the maximum shared efficiency first method;

[0012] Traverse the jobs in the job queue to determine jobs to be run and jobs to be preempted; the jobs to be run are jobs that have sufficient resources to be placed after being sorted according to priority, and the jobs to be preempted are jobs that do not have sufficient resources to be placed after being sorted according to priority;

[0013] The optimal packaging strategy and the interference prediction model of job colocation are used to determine the placement of the jobs to be run;

[0014] Place jobs and update cluster status;

[0015] Monitor new job submissions and update scheduling intervals. If a new job is submitted, jump to the step of performing interference profiling analysis on the newly submitted job. Otherwise, jump to the step of determining the job scheduling interval.

[0016] As a preferred technical solution, the interference profile analysis is performed on the newly submitted job to obtain the interference feature representation of the job and add it to the job queue, specifically:

[0017] Assign job j to the specified GPU resources exclusively for execution, and collect the throughput of job j when it is exclusively running;

[0018] Co-locate job j with each benchmark job in turn, and then get the throughput of job j after co-locating with n benchmark jobs respectively;

[0019] The throughput after co-location is collected, and job j is characterized, wherein the characteristic representation includes an interference sensitivity characteristic of job j and an influence capability characteristic of job j; the interference sensitivity characteristic of job j is the degree to which job j is affected by other jobs, and the influence capability characteristic of job j is the degree to which job j affects other loads.

[0020] As a preferred technical solution, the collection of cluster runtime information and job information is specifically as follows:

[0021] The cluster runtime information and job information are collected through the scheduling framework Volcano. Specifically, the API interfaces in the openSession function are called, including JobInfo, NodeInfo, and QueueInfo, to obtain the number of available GPUs, the number of requested GPUs, job queue information, and the number of job preemption times during cluster runtime.

[0022] As a preferred technical solution, the determination of the job scheduling interval is specifically as follows:

[0023] Defining TCG t is the total number of GPUs in the cluster at time t, TRG t is the total number of GPU requests in the queue at time t, and the scheduling interval SI at time t t The calculation method is as follows, where m1 and m2 represent the preemption cost of resource requests for jobs within a node and outside a node, respectively:

[0024]

[0025] As a preferred technical solution, the priority calculation method of job j at time t in the maximum sharing efficiency priority method is as follows:

[0026]

[0027] Among them G t is the number of GPUs currently assigned to job j by the scheduler, C t is the number of jobs co-located with job j, and CKPT is the total number of preemptions of job j.

[0028] As a preferred technical solution, the optimal packaging strategy is specifically:

[0029]

[0030] where v 1 and v 2 Represents the normalized throughput of the two jobs after bundling.

[0031] As a preferred technical solution, the interference prediction model for job colocation includes four normalized throughput prediction models, which are trained based on random forests and are:

[0032] SGCM model: predicts the normalized throughput of two single-GPU jobs after packaging them, and is trained to learn the function Γ SGCM =Φ SGCM (f 1 , f 2 ),in represents the characteristics of the j-th single GPU job, Represents the normalized throughput prediction values ​​of two single-GPU jobs after packaging;

[0033] DTCM model: predicts the normalized throughput of two multi-GPU jobs with the same resource requirements after packaging, and is trained to learn Γ DTCM =Φ DTCM (f 1 , f 2 ),in represents the characteristics of the j-th multi-GPU job, Represents the normalized throughput prediction values ​​of two multi-GPU jobs after packaging;

[0034] DTSPM model: predicts the normalized throughput of multi-GPU jobs and multi-GPU jobs after packaging multiple single-GPU jobs. It is trained to learn the function where f 1 Represents the characteristics of multi-GPU jobs, is the average of all single-GPU job characteristics, α represents the number of resources for multi-GPU jobs, and β represents the number of single-GPU jobs. It represents the predicted value of the normalized throughput after packaging a multi-GPU job with multiple single-GPU jobs.

[0035] SGSPM model: predicts the normalized throughput of a single GPU job after packaging multiple single GPU jobs and multiple single GPU jobs, and is trained to learn the function Γ SGSPM =Φ SGSPM (f 1 , f 2 ), where f 1 , f 2 , represent the characteristics of multi-GPU jobs and single-GPU jobs, respectively. Indicates the predicted value of the normalized throughput of a single GPU job after packaging.

[0036] In a second aspect, the present invention provides a GPU sharing scheduling system supporting heterogeneous job scales, which is applied to the GPU sharing scheduling method supporting heterogeneous job scales, including an interference profile analysis module, an information collection module, a scheduling interval determination module, a job sorting module, a job determination module, a job location determination module, a job placement module, and a monitoring update module;

[0037] The interference profile analysis module is used to perform interference profile analysis on a newly submitted job, obtain interference feature representation of the job and add it to the job queue;

[0038] The information collection module is used to collect cluster runtime information and job information, including the number of available GPUs in the cluster, the total number of GPUs requested by the job, the job queue, and the number of job preemption times;

[0039] The scheduling interval determination module is used to determine the job scheduling interval according to the number of available GPUs in the cluster, the number of GPUs requested by the job, and the job queue information;

[0040] The job sorting module is used to update the job priority based on the job runtime information and sort the jobs by the maximum shared efficiency first method;

[0041] The job determination module is used to traverse the jobs in the job queue to determine the jobs to be run and the jobs to be preempted; the jobs to be run are jobs that have sufficient resources to be placed after being sorted according to priority, and the jobs to be preempted are jobs that do not have sufficient resources to be placed after being sorted according to priority;

[0042] The job location determination module is used to call the optimal packaging strategy and the job co-location interference prediction model to determine the placement location of the job to be run;

[0043] The job placement module is used to place jobs and update cluster status;

[0044] The monitoring and updating module is used to monitor the submission of new jobs and update the scheduling interval. If a new job is submitted, it jumps to the interference profile analysis module, otherwise it jumps to the scheduling interval determination module.

[0045] In a third aspect, the present invention provides an electronic device, the electronic device comprising:

[0046] at least one processor; and,

[0047] a memory communicatively connected to the at least one processor; wherein,

[0048] The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the GPU sharing scheduling method that supports heterogeneous job scales.

[0049] In a fourth aspect, the present invention provides a computer-readable storage medium storing a program, characterized in that when the program is executed by a processor, the GPU sharing scheduling method supporting heterogeneous job scales is implemented.

[0050] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0051] (1) Feature representation: The present invention performs feature representation by analyzing the interference sensitivity and impact capacity of the job, which enables the system to better understand the needs and potential impact of each job and make more reasonable scheduling decisions.

[0052] (2) Prediction model: An interference prediction model is used to adapt to different task packaging modes to ensure good operating performance even in complex and changing working environments.

[0053] (3) Packaging strategy: An optimal job packaging strategy that is independent of job duration is proposed, which helps to stabilize job performance and achieve the best resource utilization status regardless of the job duration.

[0054] (4) Adaptive Scheduling: By implementing adaptive scheduling interval adjustment, this method can prioritize existing jobs under high load conditions and respond to new job requests more quickly under low load conditions, thereby reducing the additional overhead caused by job preemption.

[0055] (5) Priority sorting: Using the maximum shared efficiency priority method, jobs with higher shared efficiency are encouraged to be executed first. This not only shortens the task completion time (Job Completion Time, JCT), but also reduces the queuing waiting time of jobs.

[0056] Compared with traditional scheduling schemes, the advantage of the present invention is that it can effectively handle unpredictable deep learning workloads without relying on the estimation of task duration, thereby reducing the operating costs of GPU data centers and improving the overall job scheduling efficiency. The present invention greatly improves the resource scheduling in the GPU cluster by comprehensively considering interference characteristics, predicting job behavior, optimizing job packaging, adaptive scheduling, and priority management, achieving a significant reduction in job completion time and queuing time, thereby improving the resource utilization and economy of the entire system. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0058] Figure 1 A flowchart of a GPU sharing scheduling method supporting heterogeneous job scales according to an embodiment of the present invention;

[0059] Figure 2 A flowchart of a GPU shared scheduling operation supporting heterogeneous operation scales according to an embodiment of the present invention;

[0060] Figure 3 It is a schematic diagram of GPU sharing of heterogeneous job scales proposed by the present invention and existing GPU sharing;

[0061] Figure 4 It is a job completion time diagram of the present invention and the comparison algorithm;

[0062] Figure 5 is a graph of the number of active jobs of the present invention and the comparison algorithm;

[0063] Figure 6 It is the queue time diagram of the present invention and the comparison algorithm;

[0064] Figure 7 A block diagram of a GPU sharing scheduling system supporting heterogeneous job scales according to an embodiment of the present invention;

[0065] Figure 8 4 is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0066] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0067] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0068] See also Figure 1 , Figure 2 This embodiment provides a GPU sharing scheduling method supporting heterogeneous job scales, including the following steps:

[0069] S1. Perform interference profiling analysis on the newly submitted job, obtain the interference feature representation of the job and add it to the job queue.

[0070] Furthermore, step S1 specifically includes the following contents:

[0071] S11, the system allocates job j exclusively to the specified GPU resources for running, and collects the throughput data of job j when it runs exclusively, denoted as T j When submitting a job, the user only needs to import report_throughput(thp_value) in the model training code, where thp-value is the current job throughput value calculated by the user, which is the number of iterations per second. The report_throughput function uses the gRPC protocol to establish a communication channel between the job execution thread and the scheduler. The scheduler listens to the gRPC messages from each job execution thread. After receiving the throughput data, the system will record and store the throughput data of the job, denoted as T i .

[0072] S12: The system selects n benchmark loads and runs them with job j in turn. It also collects the throughput of the new job after being co-located with each benchmark load through the same API interface and gRPC mechanism, which is recorded as At the same time, the throughput of the benchmark load after co-location with job j is recorded and recorded as Throughput data when the benchmark workload is run exclusively T b1 、T b2 ...T bn Based on the above collected data, the system calculates the interference sensitivity characteristics and impact capability characteristics of job j. The interference sensitivity characteristics are expressed as The influencing capability characteristics are expressed as The feature of job j is expressed as The openSessionAPI interface is called through the Volcano scheduling framework to store the job features obtained through gRPC in the job information.

[0073] S2. Collect cluster runtime information and job information.

[0074] Furthermore, step S2 is specifically as follows:

[0075] The data is collected through the scheduling framework Volcano. Specifically, the APIs in the openSession function are called, including JobInfo, NodeInfo, and QueueInfo, to obtain the number of available GPUs, the number of requested GPUs, job queue information, and the number of job preemption times when the cluster is running.

[0076] Furthermore, first, the JobInfo interface is called to obtain information such as the number of GPU requests, the running status of the job, the job priority, and the number of job preemption times of all jobs in the current cluster. Secondly, the system collects information such as the number of available GPUs, the number of allocated GPUs, and the GPU utilization rate of each node in the cluster by calling the NodeInfo interface. Next, the system calls the QueueInfo interface to obtain the queued job information in the current job queue, including the resource request status of each job and its queue order in the queue.

[0077] S3. Determine the job scheduling interval based on the number of available GPUs in the cluster, the total number of GPUs requested by the job, and the job queue information.

[0078] Furthermore, step S3 is specifically as follows:

[0079] First, define TCG t Define TRG as the number of GPUs available in the cluster at time t. t is the total number of GPU requests for all jobs in the queue at time t. In order to ensure the reasonable scheduling of resources, the present invention dynamically adjusts the scheduling interval in each scheduling cycle. The scheduling interval SI t The calculation formula is as follows:

[0080]

[0081] Among them, 40 and 60 represent the preemption overhead of resource requests for jobs within the same node and jobs across nodes, respectively. When there are sufficient GPU resources available in the cluster, TRG t When the available GPU resources are small, TCG t The smaller the resource allocation is, the shorter the scheduling interval is, so as to respond to job resource requirements more quickly.

[0082] S4. Based on the job runtime information, update the job priority and sort the jobs by the maximum shared efficiency first method.

[0083] Furthermore, the priority calculation formula of job j at time t is as follows:

[0084]

[0085] Among them, G t is the number of GPUs currently assigned to job j by the scheduler, C tis the number of jobs co-located with job j, and CKPT is the total number of preemptions of job j. The priority calculation takes into account the resources currently occupied by the job, the number of other jobs sharing resources with it, and the preemption history of the job to ensure efficient resource sharing and preemption control. The jobs to be run are the jobs that have sufficient resources to be placed after being sorted according to the priority. The system first sorts all jobs according to the above priorities, and then traverses the job queue to check one by one whether there are enough GPU resources to be allocated to the current job. For jobs with sufficient resources, the system marks them as jobs to be run. On the contrary, for jobs with lower priorities that cannot obtain sufficient resources, the system marks them as jobs to be preempted. These jobs will be given priority by the scheduler for preemption to release resources for higher priority jobs to run.

[0086] S5. Traverse the jobs in the job queue to determine jobs to be run and jobs to be preempted.

[0087] It can be understood that the jobs to be run are jobs that have sufficient resources to be placed after being sorted according to priorities, and the jobs to be preempted are jobs that do not have sufficient resources to be placed after being sorted according to priorities.

[0088] S6. Call the optimal packaging strategy and the job co-location interference prediction model to determine the placement location of the job to be run.

[0089] Furthermore, the optimal packaging strategy is specifically:

[0090]

[0091] where v 1 and v 2 Represents the normalized throughput of the two jobs after packaging. This formula ensures that the throughput of the two jobs will not drop significantly when they are co-located, thereby improving resource sharing efficiency. In order to more accurately predict the interference of co-located jobs, this patent uses the interference prediction model trained by random forest in scikit-learn. For a comparison chart of the placement of heterogeneous-scale jobs and the currently supported job placement, please refer to Figure 3 , three jobs arrive at the same time, and the number of resources requested is 1, 1 and 2 respectively. Scenario 1 is that two single-GPU jobs share the same GPU, scenario 2 is that two single-GPU jobs each occupy a GPU exclusively, and scenario 3 is that a multi-GPU job occupies a GPU exclusively. The additional scenario supported by the present invention is to execute three jobs A, B and C at the same time, and job C is packaged together with jobs A and B at the same time. The three jobs share two GPUs. In order to support these scenarios, it is necessary to predict the throughput changes of the jobs after packaging in different scenarios. For this purpose, the present invention establishes four throughput prediction models, which correspond to the packaging methods of different types of jobs:

[0092] SGCM model: predicts the normalized throughput of two single-GPU jobs after packaging them, and is trained to learn the function Γ SGCM =Φ SGCM (f 1 , f 2 ),in represents the characteristics of the j-th single GPU job, Represents the normalized throughput prediction values ​​of two single-GPU jobs after packaging;

[0093] DTCM model: predicts the normalized throughput of two multi-GPU jobs with the same resource requirements after packaging, and is trained to learn Γ DTCM =Φ DTCM (f 1 , f 2 ),in represents the characteristics of the j-th multi-GPU job, Represents the normalized throughput prediction values ​​of two multi-GPU jobs after packaging;

[0094] DTSPM model: predicts the normalized throughput of multi-GPU jobs and multi-GPU jobs after packaging multiple single-GPU jobs. It is trained to learn the function where f 1 Represents the characteristics of multi-GPU jobs, is the average of all single-GPU job characteristics, α represents the number of resources for multi-GPU jobs, and β represents the number of single-GPU jobs. It represents the predicted value of the normalized throughput after packaging a multi-GPU job with multiple single-GPU jobs.

[0095] SGSPM model: predicts the normalized throughput of a single GPU job after packaging multiple single GPU jobs and multiple single GPU jobs, and is trained to learn the function Γ SGSPM =Φ SGSPM (f 1 , f 2 ), where f 1 , f 2 , represent the characteristics of multi-GPU jobs and single-GPU jobs, respectively. Indicates the predicted value of the normalized throughput of a single GPU job after packaging.

[0096] Finally, a simulation-based method is used to simulate the submission of deep learning workloads using real GPU cluster load data from SenseTime and Microsoft. Through the simulation method, the performance of the scheduling algorithm can be evaluated and compared without affecting the real cluster. The public real GPU cluster dataset is shown in Table 1, and the submitted jobs are randomly selected from the typical deep learning (DL) workloads shown in Table 2.

[0097] Table 1 Dataset characteristics

[0098]

[0099] Table 2 DL workloads used in the experiments

[0100]

[0101] Table 3 Effect of the selection of a single benchmark load on the throughput prediction accuracy (RMSE metric) of the four placement models

[0102] Benchmark load DTCM SGSPM DTSPM SGCM VGG-11 0.037 0.0889 0.047 0.109 DCGAN 0.0393 0.041 0.033 0.141 MobileNetV3 0.055 0.1104 0.0337 0.132 PNASNetB 0.0387 0.0861 0.0452 0.136 PointNet 0.0289 0.0265 0.024 0.088 PPO 0.0427 0.0741 0.0324 0.115 ResNet-18 0.0274 0.037 0.0448 0.105 LSTM 0.0449 0.0741 0.044 0.1061

[0103] Table 3 shows the impact of the selection of benchmark loads on the throughput prediction accuracy (RMSE metric) of the four placement models when the number of benchmark loads n=1. It can be seen that the prediction accuracy is the highest in the DTCM placement scenario and reaches 94.5%+. The prediction accuracy of the SGCM placement scenario is the lowest, but it also has a prediction accuracy of 85%+. In addition, different benchmark loads also have a significant impact on the prediction accuracy. Selecting PointNet as the benchmark load has the lowest prediction error. However, in the experiment, PointNet also has the highest influence capability and the lowest interference sensitivity. Simply selecting PointNet as the benchmark load will lack the evaluation of the impact of high interference capability loads on the results. Therefore, this experiment selects VGG-11 with moderate influence capability and interference sensitivity as the benchmark load.

[0104] The comparison algorithms include the following five:

[0105] (1) FIFO algorithm: widely used in systems such as Yarn and Kubernetes. Although simple, it often does not work well because it does not take into account the characteristics of job runtime.

[0106] (2) SJF algorithm: Its purpose is to minimize the job completion time (JCT) by giving priority to shorter jobs and reduce head of line blocking (HOL) through preemption. However, it is difficult to apply in practical scenarios because it requires accurate prediction of job duration.

[0107] (3) QSSF algorithm: predicts and prioritizes shorter jobs.

[0108] (4) Tiresias algorithm: Prioritizes tasks with the least service time so that short tasks can be completed faster without obtaining task duration information in advance.

[0109] (5) Lucid algorithm: A non-preemptive, sharable, data-driven approach for estimating job duration and packaging jobs based on expected performance impact.

[0110] Figure 4 The cumulative distribution function graph of the average job completion time of all algorithms on three clusters is shown. Concord is the algorithm proposed in this patent. For short-term jobs within ten minutes, the performance of all algorithms is similar. For jobs longer than ten minutes, the algorithm Concord proposed in this patent is significantly higher than the comparison algorithm. In addition, this patent supports heterogeneous job scales of job sharing GPU. In comparison, the number of jobs that Concord can accommodate with the same resources is much higher than that of the comparison algorithm. Figure 5 As shown, on Venus, Saturn, and Philly, the compared algorithms can accommodate up to 400, 800, and 1200 jobs, respectively, while Concord can accommodate up to 600, 1000, and 2000 jobs, which is a 1.25-1.6x improvement. Figure 6 The comparison of average job queueing time is shown. Concord has the lowest job queueing time on all clusters.

[0111] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0112] Based on the same idea as the GPU sharing scheduling method supporting heterogeneous job scales in the above-mentioned embodiment, the present invention also provides a GPU sharing scheduling system supporting heterogeneous job scales, which can be used to execute the above-mentioned GPU sharing scheduling method supporting heterogeneous job scales. For ease of explanation, the structural diagram of the embodiment of the GPU sharing scheduling system supporting heterogeneous job scales only shows the parts related to the embodiment of the present invention. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than the illustrated one, or combine certain components, or arrange the components differently.

[0113] See also Figure 7In another embodiment of the present application, a GPU sharing scheduling system 100 supporting heterogeneous job scales is provided, the system comprising an interference profile analysis module 101, an information collection module 102, a scheduling interval determination module 103, a job sorting module 104, a job determination module 105, a job location determination module 106, a job placement module 107, and a monitoring update module 108;

[0114] The interference profile analysis module 101 is used to perform interference profile analysis on a newly submitted job, obtain an interference feature representation of the job and add it to the job queue;

[0115] The information collection module 102 is used to collect cluster runtime information and job information, including the number of available GPUs in the cluster, the total number of GPUs requested by the job, the job queue, and the number of job preemption times;

[0116] The scheduling interval determination module 103 is used to determine the job scheduling interval according to the number of available GPUs in the cluster, the total number of GPUs requested by the job, and the job queue information;

[0117] The job sorting module 104 is used to update the job priority based on the job runtime information and sort the jobs by the maximum sharing efficiency first method;

[0118] The job determination module 105 is used to traverse the jobs in the job queue to determine the jobs to be run and the jobs to be preempted; the jobs to be run are jobs that have sufficient resources to be placed after being sorted according to the priority, and the jobs to be preempted are jobs that do not have sufficient resources to be placed after being sorted according to the priority;

[0119] The job location determination module 106 is used to call the optimal packaging strategy and the job co-location interference prediction model to determine the placement location of the job to be run;

[0120] The job placement module 107 is used to place jobs and update cluster status;

[0121] The monitoring and updating module 108 is used to monitor the submission of new jobs and update the scheduling interval. If a new job is submitted, it jumps to the interference profile analysis module, otherwise it jumps to the scheduling interval determination module.

[0122] It should be noted that the GPU shared scheduling system supporting heterogeneous job scales of the present invention corresponds one-to-one to the GPU shared scheduling method supporting heterogeneous job scales of the present invention. The technical features and beneficial effects described in the above-mentioned embodiment of the GPU shared scheduling method supporting heterogeneous job scales are applicable to the embodiment of GPU shared scheduling supporting heterogeneous job scales. For specific contents, please refer to the description in the embodiment of the method of the present invention, which will not be repeated here. This is hereby declared.

[0123] In addition, in the implementation of the GPU shared scheduling system supporting heterogeneous job scales in the above-mentioned embodiment, the logical division of each program module is only an example. In actual applications, the above-mentioned functions can be assigned to different program modules as needed, for example, for the convenience of corresponding hardware configuration requirements or software implementation. That is, the internal structure of the GPU shared scheduling system supporting heterogeneous job scales is divided into different program modules to complete all or part of the functions described above.

[0124] See also Figure 7 In one embodiment, an electronic device that implements a GPU sharing scheduling method that supports heterogeneous job scales is provided. The electronic device 200 may include a first processor 201, a first memory 202 and a bus, and may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as a GPU sharing scheduler 203 that supports heterogeneous job scales.

[0125] Among them, the first memory 202 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the first memory 202 can be an internal storage unit of the electronic device 200, such as a mobile hard disk of the electronic device 200. In other embodiments, the first memory 202 can also be an external storage device of the electronic device 200, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 200. Further, the first memory 202 can also include both an internal storage unit of the electronic device 200 and an external storage device. The first memory 202 can not only be used to store application software and various types of data installed in the electronic device 200, such as the code of the GPU sharing scheduler 203 supporting heterogeneous job scales, but also can be used to temporarily store data that has been output or is to be output.

[0126] In some embodiments, the first processor 201 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips, etc. The first processor 201 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes various functions and processes data of the electronic device 200 by running or executing programs or modules stored in the first memory 202, and calling data stored in the first memory 202.

[0127] Figure 8 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 8 The structure shown does not constitute a limitation on the electronic device 200 , and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0128] The GPU sharing scheduler 203 supporting heterogeneous job scales stored in the first memory 202 in the electronic device 200 is a combination of multiple instructions. When running in the first processor 201, it can achieve:

[0129] Perform interference profiling analysis on newly submitted jobs, obtain interference feature representations of the jobs, and add them to the job queue;

[0130] Collect cluster runtime information and job information, including the number of available GPUs in the cluster, the total number of GPUs requested by the job, the job queue, and the number of job preemption times;

[0131] Determine the job scheduling interval based on the number of available GPUs in the cluster, the total number of GPUs requested by the job, and the job queue information;

[0132] Based on the job runtime information, the job priority is updated and the jobs are sorted by the maximum shared efficiency first method;

[0133] Traverse the jobs in the job queue to determine jobs to be run and jobs to be preempted; the jobs to be run are jobs that have sufficient resources to be placed after being sorted according to priority, and the jobs to be preempted are jobs that do not have sufficient resources to be placed after being sorted according to priority;

[0134] The optimal packaging strategy and the interference prediction model of job colocation are used to determine the placement of the jobs to be run;

[0135] Place jobs and update cluster status;

[0136] Monitor new job submissions and update scheduling intervals. If a new job is submitted, jump to the step of performing interference profiling analysis on the newly submitted job. Otherwise, jump to the step of determining the job scheduling interval.

[0137] Furthermore, if the module / unit integrated in the electronic device 200 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0138] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0139] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0140] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A GPU sharing scheduling method supporting heterogeneous job scales, characterized in that: The steps include: Perform interference profiling analysis on newly submitted jobs, obtain interference feature representations of the jobs, and add them to the job queue; Collect cluster runtime information and job information, including the number of available GPUs in the cluster, the total number of GPUs requested by the job, the job queue, and the number of job preemption times; Determine the job scheduling interval based on the number of available GPUs in the cluster, the total number of GPUs requested by the job, and the job queue information; Based on the job runtime information, the job priority is updated and the jobs are sorted by the maximum shared efficiency first method; Traverse the jobs in the job queue to determine jobs to be run and jobs to be preempted; the jobs to be run are jobs that have sufficient resources to be placed after being sorted according to priority, and the jobs to be preempted are jobs that do not have sufficient resources to be placed after being sorted according to priority; The optimal packaging strategy and the interference prediction model of job colocation are used to determine the placement of the jobs to be run; Place jobs and update cluster status; Monitor new job submissions and update scheduling intervals. If a new job is submitted, jump to the step of performing interference profiling analysis on the newly submitted job. Otherwise, jump to the step of determining the job scheduling interval.

2. The GPU sharing scheduling method supporting heterogeneous job scales according to claim 1, characterized in that: The interference profile analysis is performed on the newly submitted job to obtain the interference feature representation of the job and add it to the job queue, specifically: Assign job j to the specified GPU resources exclusively for execution, and collect the throughput of job j when it is exclusively running; Co-locate job j with each benchmark job in turn, and then get the throughput of job j after co-locating with n benchmark jobs respectively; The throughput after co-location is collected, and job j is characterized, wherein the characteristic representation includes an interference sensitivity characteristic of job j and an influence capability characteristic of job j; the interference sensitivity characteristic of job j is the degree to which job j is affected by other jobs, and the influence capability characteristic of job j is the degree to which job j affects other loads.

3. The GPU sharing scheduling method supporting heterogeneous job scales according to claim 1, characterized in that: The collection of cluster runtime information and job information is specifically: The cluster runtime information and job information are collected through the scheduling framework Volcano. Specifically, the API interfaces in the openSession function are called, including JobInfo, NodeInfo, and QueueInfo, to obtain the number of available GPUs, the number of requested GPUs, job queue information, and the number of job preemption times during cluster runtime.

4. The GPU sharing scheduling method supporting heterogeneous job scales according to claim 1, characterized in that: The specific determination of the job scheduling interval is: Defining TCG t is the total number of GPUs in the cluster at time t, TRG t is the total number of GPU requests in the queue at time t, and the scheduling interval SI at time t t The calculation method is as follows, where m1 and m2 represent the preemption cost of resource requests for jobs within a node and outside a node, respectively:

5. The GPU sharing scheduling method supporting heterogeneous job scales according to claim 1, characterized in that: The priority calculation method of job j at time t in the maximum sharing efficiency priority method is as follows: Among them G t is the number of GPUs currently assigned to job j by the scheduler, C t is the number of jobs co-located with job j, and CKPT is the total number of preemptions of job j.

6. The GPU sharing scheduling method supporting heterogeneous job scales according to claim 1, characterized in that: The optimal packaging strategy is specifically: Where v1 and v2 represent the normalized throughput of the two jobs after bundling.

7. The GPU sharing scheduling method supporting heterogeneous job scales according to claim 1, characterized in that: The interference prediction model for job colocation includes four normalized throughput prediction models, which are trained based on random forests and are: SGCM model: predicts the normalized throughput of two single-GPU jobs after packaging them, and is trained to learn the function Γ SGCM =Φ SGCM (f1,f2), where represents the characteristics of the j-th single GPU job, Represents the normalized throughput prediction values ​​of two single-GPU jobs after packaging; DTCM model: predicts the normalized throughput of two multi-GPU jobs with the same resource requirements after packaging, and is trained to learn Γ DTCM =Φ DTCM (f1,f2), where represents the characteristics of the j-th multi-GPU job, Represents the normalized throughput prediction values ​​of two multi-GPU jobs after packaging; DTSPM model: predicts the normalized throughput of multi-GPU jobs and multi-GPU jobs after packaging multiple single-GPU jobs. It is trained to learn the function Where f1 represents the characteristics of multi-GPU operations, is the average of all single-GPU job characteristics, a represents the number of multi-GPU job resources, β represents the number of single-GPU jobs, It represents the predicted value of the normalized throughput after packaging a multi-GPU job with multiple single-GPU jobs. SGSPM model: predicts the normalized throughput of a single GPU job after packaging multiple single GPU jobs and multiple single GPU jobs, and is trained to learn the function Γ SGSPM =Φ SGSPM (f1,f2), where f1 and f2 represent the characteristics of multi-GPU jobs and single-GPU jobs, respectively. Indicates the predicted value of the normalized throughput of a single GPU job after packaging.

8. GPU sharing scheduling system supporting heterogeneous job scales, characterized by: A GPU sharing scheduling method supporting heterogeneous job scales applied to any one of claims 1-7, comprising an interference profile analysis module, an information collection module, a scheduling interval determination module, a job sorting module, a job determination module, a job location determination module, a job placement module, and a monitoring update module; The interference profile analysis module is used to perform interference profile analysis on a newly submitted job, obtain interference feature representation of the job and add it to the job queue; The information collection module is used to collect cluster runtime information and job information, including the number of available GPUs in the cluster, the total number of GPUs requested by the job, the job queue, and the number of job preemption times; The scheduling interval determination module is used to determine the job scheduling interval according to the number of available GPUs in the cluster, the number of GPUs requested by the job, and the job queue information; The job sorting module is used to update the job priority based on the job runtime information and sort the jobs by the maximum shared efficiency first method; The job determination module is used to traverse the jobs in the job queue to determine the jobs to be run and the jobs to be preempted; the jobs to be run are jobs that have sufficient resources to be placed after being sorted according to priority, and the jobs to be preempted are jobs that do not have sufficient resources to be placed after being sorted according to priority; The job location determination module is used to call the optimal packaging strategy and the job co-location interference prediction model to determine the placement location of the job to be run; The job placement module is used to place jobs and update cluster status; The monitoring and updating module is used to monitor the submission of new jobs and update the scheduling interval. If a new job is submitted, it jumps to the interference profile analysis module, otherwise it jumps to the scheduling interval determination module.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the GPU sharing scheduling method supporting heterogeneous job scales as described in any one of claims 1-7.

10. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the GPU sharing scheduling method supporting heterogeneous job scales described in any one of claims 1-7 is implemented.

Images