GPU sharing scheduling method and device supporting heterogeneous operation scale

Abstract

The present invention discloses a GPU sharing scheduling method and device supporting heterogeneous job scales, comprising: performing interference profile analysis on newly submitted jobs, obtaining interference feature representations of the jobs and adding them to a job queue; collecting cluster runtime information and job information; determining a job scheduling interval; updating job priorities based on the job runtime information and sorting the jobs using a maximum sharing efficiency priority method; traversing the jobs in the job queue to determine jobs to be run and jobs to be preempted; calling an interference prediction model based on an optimal packaging strategy and job colocation to determine the placement of jobs to be run; placing jobs and updating cluster status; monitoring new job submissions and updating the scheduling interval; if a new job is submitted, jumping to the step of performing interference profile analysis on the newly submitted job, otherwise jumping to the step of determining the job scheduling interval. The present invention can effectively reduce job completion time and queuing time, and significantly improve the resource scheduling efficiency of the GPU cluster.

Description

Technical Field

[0001] The present invention belongs to the technical field of GPU scheduling, and in particular relates to a GPU sharing scheduling method and device supporting heterogeneous job scales. Background Art

[0002] With the rapid development of deep learning (DL) technology in recent years, it has been widely applied to numerous fields, including image classification and recommendation systems. To further enhance the capabilities of deep learning models, numerous information technology companies and research institutions have invested significant resources in building extensive deep learning infrastructure. The rise of large-scale language models (LLMs), in particular, has significantly driven demand for GPU computing power. This has also necessitated the critical role of cluster schedulers in optimizing resource allocation and coordinating various deep learning tasks.

[0003] Public cluster data records show significant resource contention between single-GPU and multi-GPU workloads. Although workloads encompass both single-GPU and multi-GPU tasks, most training jobs tend to be executed using a single GPU. Conversely, while multi-GPU jobs are fewer in number, they typically take longer to complete. Current scheduling mechanisms often prioritize multi-GPU tasks, which often limits resource availability for single-GPU jobs and leads to imbalanced resource allocation.

[0004] One possible mitigation solution is to enable multiple deep learning jobs to run together on the same GPU. However, current methods are unable to cope with resource competition between multi-GPU and single-GPU jobs, especially in effectively sharing resources between tasks with different GPU requirements. In addition, existing methods overly rely on experience-based packaging strategies, which are inadequate when faced with complex workload interference patterns. Therefore, scheduling multi-GPU and single-GPU jobs simultaneously in an environment that supports GPU sharing is a challenging task, and current schedulers perform poorly in this regard, resulting in suboptimal performance and efficiency of the entire system. Summary of the Invention

[0005] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology and provide a GPU sharing scheduling method and device that supports heterogeneous job scales. By analyzing task behavior in various interference environments, the problems caused by the co-location of multi-GPU and single-GPU workloads are solved.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a GPU sharing scheduling method supporting heterogeneous job scales, comprising the following steps:

[0008] Perform interference profiling analysis on newly submitted jobs, obtain interference feature representations of the jobs, and add them to the job queue;

[0009] Collect cluster runtime information and job information, including the number of available GPUs in the cluster, the total number of GPUs requested by jobs, job queues, and job preemption times;

[0010] The job scheduling interval is determined based on the number of available GPUs in the cluster, the total number of GPUs requested by the job, and the job queue information;

[0011] Based on job runtime information, job priorities are updated and jobs are sorted using the maximum shared efficiency first method;

[0012] Traverse the jobs in the job queue to determine jobs to be run and jobs to be preempted; the jobs to be run are jobs that have sufficient resources to be placed after sorting by priority, and the jobs to be preempted are jobs that do not have sufficient resources to be placed after sorting by priority;

[0013] Determine the placement of pending jobs by invoking the optimal packaging strategy and the job colocation interference prediction model;

[0014] Place jobs and update cluster status;

[0015] Monitor new job submissions and update scheduling intervals. If a new job is submitted, jump to the step of performing interference profiling analysis on the newly submitted job. Otherwise, jump to the step of determining the job scheduling interval.

[0016] As a preferred technical solution, the interference profile analysis is performed on the newly submitted job to obtain the interference feature representation of the job and add it to the job queue, specifically:

[0017] Assign job j exclusively to the specified GPU resources and collect the throughput of job j when it is running exclusively.

[0018] Collocate job j with each benchmark job in turn, and then obtain the throughput of job j after collocating with n benchmark jobs respectively;

[0019] The throughput after co-location is collected, and job j is characterized. The characteristic representation includes the interference sensitivity characteristic of job j and the influence capability characteristic of job j. The interference sensitivity characteristic of job j is the degree to which job j is affected by other jobs, and the influence capability characteristic of job j is the degree to which job j affects other loads.

[0020] As a preferred technical solution, the collection of cluster runtime information and job information is specifically as follows:

[0021] Cluster runtime information and job information are collected through the Volcano scheduling framework. Specifically, the API interfaces in the openSession function are called, including JobInfo, NodeInfo, and QueueInfo, to obtain the number of available GPUs, the number of requested GPUs, job queue information, and the number of job preemption times during cluster runtime.

[0022] As a preferred technical solution, the job scheduling interval is determined as follows:

[0023] Defining TCG t is the total number of GPUs in the cluster at time t, TRG t is the total number of GPU requests in the queue at time t, and the scheduling interval SI at time t t The calculation method is as follows, where m1 and m2 represent the preemption cost of resource requests for jobs within a node and outside a node, respectively:

[0024]

[0025] As a preferred technical solution, the priority calculation method of job j at time t in the maximum sharing efficiency priority method is as follows:

[0026]

[0027] Among them G t is the number of GPUs currently assigned to job j by the scheduler, C t is the number of jobs co-located with job j, and CKPT is the total number of preemptions of job j.

[0028] As a preferred technical solution, the optimal packaging strategy is specifically as follows:

[0029]

[0030] Where v1 and v2 represent the normalized throughput of the two jobs after bundling.

[0031] As a preferred technical solution, the interference prediction model for job colocation includes four normalized throughput prediction models, which are trained based on random forests and are:

[0032] SGCM model: predicts the normalized throughput of two jobs after packaging two single-GPU jobs, and is trained to learn the function Γ SGCM =Φ SGCM (f1, f2), where represents the characteristics of the j-th single GPU job, Represents the normalized throughput prediction value of two single-GPU jobs after packaging;

[0033] DTCM model: predicts the normalized throughput of two multi-GPU jobs with the same resource requirements after packaging, and is trained to learn Γ DTCM =Φ DTCM (f1, f2), where represents the characteristics of the j-th multi-GPU job, Represents the normalized throughput prediction value of two multi-GPU jobs after packaging;

[0034] DTSPM model: predicts the normalized throughput of multi-GPU jobs and multi-GPU jobs after packaging multiple single-GPU jobs. It is trained to learn the function Where f1 represents the characteristics of multi-GPU operations, is the average value of all single-GPU job characteristics, α represents the number of resources for multi-GPU jobs, and β represents the number of single-GPU jobs. The predicted value of the normalized throughput of a multi-GPU job and multiple single-GPU jobs packaged together.

[0035] SGSPM model: predicts the normalized throughput of a multi-GPU job and a single-GPU job after packaging multiple single-GPU jobs. It is trained to learn the function Γ SGSPM =Φ SGSPM (f1, f2), where f1 and f2 represent the characteristics of multi-GPU jobs and single-GPU jobs, respectively. Indicates the predicted value of the normalized throughput of a single GPU job after packaging.

[0036] In a second aspect, the present invention provides a GPU shared scheduling system supporting heterogeneous job scales, which is applied to the GPU shared scheduling method supporting heterogeneous job scales, including an interference profile analysis module, an information collection module, a scheduling interval determination module, a job sorting module, a job determination module, a job location determination module, a job placement module, and a monitoring and update module;

[0037] The interference profile analysis module is used to perform interference profile analysis on newly submitted jobs, obtain interference feature representations of the jobs, and add them to the job queue;

[0038] The information collection module is used to collect cluster runtime information and job information, including the number of available GPUs in the cluster, the total number of GPUs requested by the job, the job queue, and the number of job preemptions;

[0039] The scheduling interval determination module is used to determine the job scheduling interval based on the number of available GPUs in the cluster, the total number of GPUs requested by the job, and the job queue information;

[0040] The job sorting module is used to update the job priority based on the job runtime information and sort the jobs by the maximum shared efficiency first method;

[0041] The job determination module is used to traverse the jobs in the job queue and determine jobs to be run and jobs to be preempted; the jobs to be run are jobs that have sufficient resources to be placed after being sorted according to priority, and the jobs to be preempted are jobs that do not have sufficient resources to be placed after being sorted according to priority;

[0042] The job location determination module is used to call the optimal packaging strategy and the job co-location interference prediction model to determine the placement location of the job to be run;

[0043] The job placement module is used to place jobs and update cluster status;

[0044] The monitoring and updating module is used to monitor the submission of new jobs and update the scheduling interval. If a new job is submitted, it jumps to the interference portrait analysis module; otherwise, it jumps to the scheduling interval determination module.

[0045] In a third aspect, the present invention provides an electronic device, comprising:

[0046] at least one processor; and,

[0047] a memory communicatively connected to the at least one processor; wherein,

[0048] The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to execute the GPU sharing scheduling method that supports heterogeneous job scales.

[0049] In a fourth aspect, the present invention provides a computer-readable storage medium storing a program, characterized in that when the program is executed by a processor, the GPU sharing scheduling method supporting heterogeneous job scales is implemented.

[0050] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0051] (1) Feature representation: The present invention performs feature representation by analyzing the interference sensitivity and impact capability of the job, which enables the system to better understand the needs and potential impact of each job and thus make more reasonable scheduling decisions.

[0052] (2) Prediction model: An interference prediction model is used to adapt to different task packaging modes to ensure good operating performance even in complex and changing working environments.

[0053] (3) Packaging strategy: An optimal job packaging strategy that is independent of job duration is proposed, which helps to stabilize job performance and achieve optimal resource utilization regardless of job duration.

[0054] (4) Adaptive Scheduling: By implementing adaptive scheduling interval adjustment, this method can prioritize existing jobs under high-load conditions and respond to new job requests more quickly under low-load conditions, thereby reducing the additional overhead caused by job preemption.

[0055] (5) Priority sorting: Using the maximum sharing efficiency priority method, jobs with higher sharing efficiency are encouraged to be executed first. This not only shortens the task completion time (Job Completion Time, JCT), but also reduces the queuing waiting time of jobs.

[0056] Compared to traditional scheduling schemes, this invention's advantage lies in its ability to effectively handle unpredictable deep learning workloads without relying on task duration estimates, thereby reducing GPU data center operating costs and improving overall job scheduling efficiency. By comprehensively considering interference characteristics, predicting job behavior, optimizing job packaging, adaptive scheduling, and priority management, this invention significantly improves resource scheduling in GPU clusters, significantly reducing job completion and queuing times, and thus improving resource utilization and economic efficiency of the entire system. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0058] Figure 1 This is a flowchart of a GPU sharing scheduling method supporting heterogeneous job scales according to an embodiment of the present invention;

[0059] Figure 2 This is a flowchart of an embodiment of the present invention that supports GPU shared scheduling of heterogeneous job scales;

[0060] Figure 3 Schematic diagram of GPU sharing of heterogeneous job scales proposed by the present invention and existing GPU sharing;

[0061] Figure 4 It is a graph of the job completion time of the present invention and the comparison algorithm;

[0062] Figure 5 is a graph of the number of active jobs for the present invention and the comparison algorithm;

[0063] Figure 6 It is the queue time graph of the present invention and the comparison algorithm;

[0064] Figure 7 A block diagram of a GPU sharing scheduling system supporting heterogeneous job scales according to an embodiment of the present invention;

[0065] Figure 8 2 is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0066] In order to enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0067] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0068] See also Figure 1 、 Figure 2 This embodiment provides a GPU sharing scheduling method supporting heterogeneous job scales, including the following steps:

[0069] S1. Perform interference profiling analysis on the newly submitted job, obtain the interference feature representation of the job and add it to the job queue.

[0070] Furthermore, step S1 specifically includes the following contents:

[0071] S11, the system allocates job j exclusively to the specified GPU resource for execution, and collects the throughput data of job j during exclusive execution, which is recorded as T jWhen submitting a job, the user only needs to import report_throughput(thp_value) in their model training code, where thp-value is the current job throughput value calculated by the user, which is the number of iterations per second. The report_throughput function uses the gRPC protocol to establish a communication channel between the job execution thread and the scheduler. The scheduler listens to the gRPC messages from each job execution thread. After receiving the throughput data, the system will record and store the throughput data of the job, which is recorded as T i .

[0072] S12: The system selects n benchmark workloads and runs them together with job j in turn. The system collects the throughput of the new job after being co-located with each benchmark workload through the same API interface and gRPC mechanism, which is recorded as At the same time, the throughput of the benchmark workload and job j after co-location is recorded and recorded as Throughput data T when the benchmark workload is exclusively running b1 、T b2 ...T bn Based on the above collected data, the system calculates the interference sensitivity characteristics and impact capability characteristics of job j. The interference sensitivity characteristics are expressed as The influence capability characteristic is expressed as The feature of job j is represented as The openSessionAPI interface is called through the Volcano scheduling framework to store the job features obtained through gRPC in the job information.

[0073] S2. Collect cluster runtime information and job information.

[0074] Furthermore, step S2 is specifically as follows:

[0075] Data is collected through the Volcano scheduling framework. Specifically, the APIs in the openSession function are called, including JobInfo, NodeInfo, and QueueInfo, to obtain the number of available GPUs, the number of requested GPUs, job queue information, and the number of job preemptions during cluster runtime.

[0076] Furthermore, the system first calls the JobInfo interface to obtain information such as the number of GPU requests, job running status, job priority, and job preemption count for all jobs in the current cluster. Secondly, the system calls the NodeInfo interface to collect information such as the number of available GPUs, the number of allocated GPUs, and GPU utilization for each node in the cluster. Next, the system calls the QueueInfo interface to obtain information about the queued jobs in the current job queue, including each job's resource request status and its queue order.

[0077] S3. Determine the job scheduling interval based on the number of available GPUs in the cluster, the total number of GPUs requested by the job, and the job queue information.

[0078] Furthermore, step S3 is specifically as follows:

[0079] First, define TCG t Define TRG as the number of GPUs available in the cluster at time t. t is the total number of GPU requests for all jobs in the queue at time t. In order to ensure the reasonable scheduling of resources, the present invention dynamically adjusts the scheduling interval in each scheduling cycle. The scheduling interval SI t The calculation formula is as follows:

[0080]

[0081] Among them, 40 and 60 represent the preemption overhead of resource requests for jobs within the same node and jobs across nodes, respectively. When the available GPU resources in the cluster are sufficient, TRG t When the GPU resources are small, the scheduling interval is relatively long to reduce unnecessary scheduling operations. t The smaller the resource allocation time, the shorter the scheduling interval, so as to respond more quickly to job resource requirements.

[0082] S4. Based on the job runtime information, update the job priority and sort the jobs by the maximum shared efficiency first method.

[0083] Furthermore, the priority calculation formula of job j at time t is as follows:

[0084]

[0085] Among them, G t is the number of GPUs currently assigned to job j by the scheduler, C tis the number of jobs co-located with job j, and CKPT is the total number of preemptions for job j. The priority calculation takes into account the resources currently occupied by the job, the number of other jobs sharing resources with it, and the preemption history of the job to ensure efficient resource sharing and preemption control. Jobs to be run are jobs that have sufficient resources to be placed after being sorted by priority. The system first sorts all jobs according to the above priorities, and then traverses the job queue, checking one by one whether there are enough GPU resources to be allocated to the current job. For jobs with sufficient resources, the system marks them as jobs to be run. Conversely, for jobs with lower priority that cannot obtain sufficient resources, the system marks them as jobs to be preempted. These jobs will be given priority by the scheduler for preemption to free up resources for higher priority jobs to run.

[0086] S5. Traverse the jobs in the job queue to determine jobs to be run and jobs to be preempted.

[0087] It is understandable that the jobs to be run are jobs that have sufficient resources to be placed after being sorted according to priority, and the jobs to be preempted are jobs that do not have sufficient resources to be placed after being sorted according to priority.

[0088] S6. Call the optimal packaging strategy and the job co-location interference prediction model to determine the placement location of the job to be run.

[0089] Furthermore, the optimal packaging strategy is specifically as follows:

[0090]

[0091] Where v1 and v2 represent the normalized throughput of the two jobs after packaging. This formula ensures that the throughput of the two jobs will not drop significantly when running co-located, thereby improving resource sharing efficiency. To more accurately predict the interference of co-located jobs, this patent uses an interference prediction model trained with random forests in scikit-learn. For a comparison chart of the placement of heterogeneous-scale jobs and currently supported job placement, please refer to Figure 3 , three jobs arrive at the same time, and the number of resources requested is 1, 1, and 2 respectively. Scenario 1 is that two single-GPU jobs share the same GPU, scenario 2 is that two single-GPU jobs each occupy a GPU exclusively, and scenario 3 is that a multi-GPU job occupies a GPU exclusively. The present invention additionally supports the scenario of executing three jobs A, B, and C simultaneously, where job C is packaged together with jobs A and B, and the three jobs share two GPUs. To support these scenarios, it is necessary to predict the throughput changes of the jobs after packaging in different scenarios. To this end, the present invention establishes four throughput prediction models, corresponding to the packaging methods of different types of jobs:

[0092] SGCM model: predicts the normalized throughput of two jobs after packaging two single-GPU jobs, and is trained to learn the function Γ SGCM =Φ SGCM (f1, f2), where represents the characteristics of the j-th single GPU job, Represents the normalized throughput prediction value of two single-GPU jobs after packaging;

[0093] DTCM model: predicts the normalized throughput of two multi-GPU jobs with the same resource requirements after packaging, and is trained to learn Γ DTCM =Φ DTCM (f1, f2), where represents the characteristics of the j-th multi-GPU job, Represents the normalized throughput prediction value of two multi-GPU jobs after packaging;

[0094] DTSPM model: predicts the normalized throughput of multi-GPU jobs and multi-GPU jobs after packaging multiple single-GPU jobs. It is trained to learn the function Where f1 represents the characteristics of multi-GPU operations, is the average value of all single-GPU job characteristics, α represents the number of resources for multi-GPU jobs, and β represents the number of single-GPU jobs. The predicted value of the normalized throughput of a multi-GPU job and multiple single-GPU jobs packaged together.

[0095] SGSPM model: predicts the normalized throughput of a multi-GPU job and a single-GPU job after packaging multiple single-GPU jobs. It is trained to learn the function Γ SGSPM =Φ SGSPM (f1, f2), where f1 and f2 represent the characteristics of multi-GPU jobs and single-GPU jobs, respectively. Indicates the predicted value of the normalized throughput of a single GPU job after packaging.

[0096] Finally, a simulation-based approach is used to simulate the submission of deep learning workloads using real GPU cluster load data from SenseTime and Microsoft. Through simulation, the performance of the scheduling algorithm can be evaluated and compared without affecting the real cluster. The public real GPU cluster dataset is shown in Table 1. The submitted jobs are randomly selected from the typical deep learning (DL) workloads shown in Table 2.

[0097] Table 1 Dataset characteristics

[0098]

[0099] Table 2 DL workloads used in the experiments

[0100]

[0101] Table 3 Effect of the selection of a single benchmark load on the throughput prediction accuracy (RMSE metric) of the four placement models

[0102] Benchmark load DTCM SGSPM DTSPM SGCM VGG-11 0.037 0.0889 0.047 0.109 DCGAN 0.0393 0.041 0.033 0.141 MobileNetV3 0.055 0.1104 0.0337 0.132 PNASNetB 0.0387 0.0861 0.0452 0.136 PointNet 0.0289 0.0265 0.024 0.088 PPO 0.0427 0.0741 0.0324 0.115 ResNet-18 0.0274 0.037 0.0448 0.105 LSTM 0.0449 0.0741 0.044 0.1061

[0103] Table 3 shows the impact of the benchmark load selection on the throughput prediction accuracy (RMSE metric) of the four placement models when the number of benchmark loads n = 1. The DTCM placement scenario achieves the highest prediction accuracy, reaching over 94.5%. The SGCM placement scenario achieves the lowest prediction accuracy, but still reaches over 85%. Different benchmark loads also significantly affect prediction accuracy. PointNet, chosen as the benchmark load, has the lowest prediction error. However, in the experiment, PointNet also has the highest influence and the lowest interference sensitivity. Simply selecting PointNet as the benchmark load will miss the impact of high-interference loads on the results. Therefore, this experiment selected VGG-11, which has moderate influence and interference sensitivity, as the benchmark load.

[0104] The comparison algorithms include the following five:

[0105] (1) FIFO algorithm: widely used in systems such as Yarn and Kubernetes. Although simple, it often does not work well because it does not consider the characteristics of job runtime.

[0106] (2) SJF algorithm: Its purpose is to minimize the job completion time (JCT) by prioritizing shorter jobs and reduce head of line blocking (HOL) through preemption. However, it is difficult to apply in practical scenarios because it requires accurate prediction of job duration.

[0107] (3) QSSF algorithm: predicts and prioritizes shorter jobs.

[0108] (4) Tiresias algorithm: prioritizes tasks with the least service time so that short tasks can be completed faster without obtaining task duration information in advance.

[0109] (5) Lucid algorithm: A non-preemptive, sharable, data-driven approach for estimating job duration and packaging jobs based on expected performance impact.

[0110] Figure 4The cumulative distribution function graph of the average job completion time of all algorithms on three clusters is shown. Concord is the algorithm proposed in this patent. For short-term jobs within ten minutes, all algorithms have similar performance. For jobs longer than ten minutes, Concord, the algorithm proposed in this patent, is significantly higher than the comparison algorithm. In addition, this patent supports the sharing of GPUs for jobs of heterogeneous job sizes. In comparison, with the same resources, Concord can accommodate far more jobs than the comparison algorithm. Figure 5 As shown in the figure, the compared algorithms can accommodate up to 400, 800, and 1200 jobs on Venus, Saturn, and Philly, respectively, while Concord can accommodate up to 600, 1000, and 2000 jobs, respectively, with an improvement of 1.25-1.6x. Figure 6 The comparison of average job queueing time is shown. Concord has the lowest job queueing time on all clusters.

[0111] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0112] Based on the same concept as the GPU sharing scheduling method supporting heterogeneous job scales in the above-mentioned embodiment, the present invention also provides a GPU sharing scheduling system supporting heterogeneous job scales, which can be used to execute the above-mentioned GPU sharing scheduling method supporting heterogeneous job scales. For ease of explanation, the structural diagram of the embodiment of the GPU sharing scheduling system supporting heterogeneous job scales only shows the parts related to the embodiment of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation of the device, and it may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0113] See also Figure 7 In another embodiment of the present application, a GPU shared scheduling system 100 supporting heterogeneous job scales is provided, the system comprising an interference profile analysis module 101, an information collection module 102, a scheduling interval determination module 103, a job sorting module 104, a job determination module 105, a job location determination module 106, a job placement module 107, and a monitoring update module 108;

[0114] The interference profile analysis module 101 is used to perform interference profile analysis on a newly submitted job, obtain an interference feature representation of the job, and add the job to the job queue;

[0115] The information collection module 102 is used to collect cluster runtime information and job information, including the number of available GPUs in the cluster, the total number of GPUs requested by the job, the job queue, and the number of job preemptions;

[0116] The scheduling interval determination module 103 is used to determine the job scheduling interval according to the number of available GPUs in the cluster, the total number of GPUs requested by the job, and the job queue information;

[0117] The job sorting module 104 is used to update the job priority based on the job runtime information and sort the jobs by the maximum shared efficiency first method;

[0118] The job determination module 105 is used to traverse the jobs in the job queue and determine jobs to be run and jobs to be preempted; the jobs to be run are jobs that have sufficient resources to be placed after being sorted according to priority, and the jobs to be preempted are jobs that do not have sufficient resources to be placed after being sorted according to priority;

[0119] The job location determination module 106 is configured to determine the placement location of the job to be run by invoking the optimal packaging strategy and the job co-location interference prediction model;

[0120] The job placement module 107 is used to place jobs and update cluster status;

[0121] The monitoring and updating module 108 is used to monitor the submission of new jobs and update the scheduling interval. If a new job is submitted, the module jumps to the interference profile analysis module; otherwise, the module jumps to the scheduling interval determination module.

[0122] It should be noted that the GPU shared scheduling system supporting heterogeneous job scales of the present invention corresponds one-to-one to the GPU shared scheduling method supporting heterogeneous job scales of the present invention. The technical features and beneficial effects described in the above-mentioned embodiment of the GPU shared scheduling method supporting heterogeneous job scales are applicable to the embodiment of the GPU shared scheduling supporting heterogeneous job scales. For specific contents, please refer to the description in the embodiment of the method of the present invention. No further details will be given here. This is hereby declared.

[0123] In addition, in the implementation of the GPU shared scheduling system supporting heterogeneous job scales in the above embodiment, the logical division of each program module is only an example. In actual applications, the above functions can be assigned to different program modules as needed, for example, for the convenience of corresponding hardware configuration requirements or software implementation. That is, the internal structure of the GPU shared scheduling system supporting heterogeneous job scales is divided into different program modules to complete all or part of the functions described above.

[0124] See also Figure 7In one embodiment, an electronic device that implements a GPU shared scheduling method that supports heterogeneous job scales is provided. The electronic device 200 may include a first processor 201, a first memory 202 and a bus, and may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as a GPU shared scheduler 203 that supports heterogeneous job scales.

[0125] The first memory 202 includes at least one type of readable storage medium, including flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the first memory 202 may be an internal storage unit of the electronic device 200, such as a mobile hard disk of the electronic device 200. In other embodiments, the first memory 202 may also be an external storage device of the electronic device 200, such as a plug-in mobile hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 200. Furthermore, the first memory 202 may include both an internal storage unit of the electronic device 200 and an external storage device. The first memory 202 can be used not only to store application software and various types of data installed in the electronic device 200, such as the code of the GPU shared scheduler 203 that supports heterogeneous job scales, but also to temporarily store data that has been output or is about to be output.

[0126] In some embodiments, the first processor 201 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including a combination of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 201 is the control core (Control Unit) of the electronic device, connecting the various components of the entire electronic device using various interfaces and lines, and executing or executing programs or modules stored in the first memory 202, as well as calling data stored in the first memory 202, to perform various functions of the electronic device 200 and process data.

[0127] Figure 8 Only the electronic device with components is shown, and it can be understood by those skilled in the art that Figure 8The structure shown does not constitute a limitation on the electronic device 200 , and the electronic device 200 may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0128] The GPU sharing scheduler 203 supporting heterogeneous job scales stored in the first memory 202 of the electronic device 200 is a combination of multiple instructions. When executed in the first processor 201, it can achieve the following:

[0129] Perform interference profiling analysis on newly submitted jobs, obtain interference feature representations of the jobs, and add them to the job queue;

[0130] Collect cluster runtime information and job information, including the number of available GPUs in the cluster, the total number of GPUs requested by jobs, job queues, and job preemption times;

[0131] The job scheduling interval is determined based on the number of available GPUs in the cluster, the total number of GPUs requested by the job, and the job queue information;

[0132] Based on job runtime information, job priorities are updated and jobs are sorted using the maximum shared efficiency first method;

[0133] Traverse the jobs in the job queue to determine jobs to be run and jobs to be preempted; the jobs to be run are jobs that have sufficient resources to be placed after sorting by priority, and the jobs to be preempted are jobs that do not have sufficient resources to be placed after sorting by priority;

[0134] Determine the placement of pending jobs by invoking the optimal packaging strategy and the job colocation interference prediction model;

[0135] Place jobs and update cluster status;

[0136] Monitor new job submissions and update scheduling intervals. If a new job is submitted, jump to the step of performing interference profiling analysis on the newly submitted job. Otherwise, jump to the step of determining the job scheduling interval.

[0137] Furthermore, if the modules / units integrated in the electronic device 200 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0138] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0139] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0140] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A GPU sharing scheduling method that supports heterogeneous job scales, characterized by: The steps include: Perform interference profiling analysis on newly submitted jobs, obtain the interference feature representation of the job, and add it to the job queue. Specifically: Assign job j exclusively to the specified GPU resources and collect the throughput of job j when it is running exclusively. Collocate job j with each benchmark job in turn, and then obtain the throughput of job j after collocating with n benchmark jobs respectively; Collect the post-colocation throughput and characterize job j. The characterization includes job j's interference sensitivity and its impact capability. The interference sensitivity of job j represents the degree to which job j is affected by other jobs, and the impact capability of job j represents the degree to which job j affects other loads. Collect cluster runtime information and job information, including the number of available GPUs in the cluster, the total number of GPUs requested by jobs, job queues, and job preemption times; The job scheduling interval is determined based on the number of available GPUs in the cluster, the total number of GPUs requested by the job, and the job queue information; Based on job runtime information, job priorities are updated and jobs are sorted using the maximum shared efficiency first method; Traverse the jobs in the job queue to determine jobs to be run and jobs to be preempted; the jobs to be run are jobs that have sufficient resources to be placed after sorting by priority, and the jobs to be preempted are jobs that do not have sufficient resources to be placed after sorting by priority; Determine the placement of pending jobs by invoking the optimal packaging strategy and the job colocation interference prediction model; Place jobs and update cluster status; Monitor new job submissions and update scheduling intervals. If a new job is submitted, jump to the step of performing interference profiling analysis on the newly submitted job. Otherwise, jump to the step of determining the job scheduling interval.

2. The GPU sharing scheduling method supporting heterogeneous job scales according to claim 1, characterized in that: The collection of cluster runtime information and job information is specifically: Cluster runtime information and job information are collected through the Volcano scheduling framework. Specifically, the API interfaces in the openSession function are called, including JobInfo, NodeInfo, and QueueInfo, to obtain the number of available GPUs, the number of requested GPUs, job queue information, and the number of job preemption times during cluster runtime.

3. The GPU sharing scheduling method supporting heterogeneous job scales according to claim 1, characterized in that: The specific determination of the job scheduling interval is: Defining TCG t is the total number of GPUs in the cluster at time t, TRG t is the total number of GPU requests in the queue at time t, and the scheduling interval SI at time t t The calculation method is as follows, where m1 and m2 represent the preemption cost of resource requests for jobs within a node and outside a node, respectively:

4. The GPU sharing scheduling method supporting heterogeneous job scales according to claim 1, characterized in that: The priority of job j at time t in the maximum shared efficiency priority method is calculated as follows: Among them G t is the number of GPUs currently assigned to job j by the scheduler, C t is the number of jobs co-located with job j, and CKPT is the total number of preemptions of job j.

5. The GPU sharing scheduling method supporting heterogeneous job scales according to claim 1, characterized in that: The optimal packaging strategy is specifically: Where v1 and v2 represent the normalized throughput of the two jobs after bundling.

6. The GPU sharing scheduling method supporting heterogeneous job scales according to claim 1, characterized in that: The interference prediction model for job colocation includes four normalized throughput prediction models, which are trained based on random forests and are: SGCM model: predicts the normalized throughput of two jobs after packaging two single-GPU jobs, and is trained to learn the function Γ SGCM =Φ SGCM (f1,f2), where represents the characteristics of the j-th single GPU job, Represents the normalized throughput prediction value of two single-GPU jobs after packaging; DTCM model: predicts the normalized throughput of two multi-GPU jobs with the same resource requirements after packaging, and is trained to learn Γ DTCM =Φ DTCM (f1,f2), where represents the characteristics of the j-th multi-GPU job, Represents the normalized throughput prediction value of two multi-GPU jobs after packaging; DTSPM model: predicts the normalized throughput of multi-GPU jobs and multi-GPU jobs after packaging multiple single-GPU jobs. It is trained to learn the function Where f1 represents the characteristics of multi-GPU operations, is the average value of all single-GPU job characteristics, α represents the number of resources for multi-GPU jobs, and β represents the number of single-GPU jobs. The predicted value of the normalized throughput of a multi-GPU job and multiple single-GPU jobs packaged together. SGSPM model: predicts the normalized throughput of a multi-GPU job and a single-GPU job after packaging multiple single-GPU jobs. It is trained to learn the function Γ SGSPM =Φ SGSPM (f1,f2), where f1 and f2 represent the characteristics of multi-GPU jobs and single-GPU jobs, respectively. Indicates the predicted value of the normalized throughput of a single GPU job after packaging.

7. GPU sharing scheduling system that supports heterogeneous job scales, characterized by: A GPU shared scheduling method supporting heterogeneous job scales, applied to any one of claims 1-6, comprising an interference profile analysis module, an information collection module, a scheduling interval determination module, a job sorting module, a job determination module, a job location determination module, a job placement module, and a monitoring and updating module; The interference profile analysis module is used to perform interference profile analysis on newly submitted jobs, obtain interference feature representations of the jobs, and add them to the job queue; The information collection module is used to collect cluster runtime information and job information, including the number of available GPUs in the cluster, the total number of GPUs requested by the job, the job queue, and the number of job preemption times; The scheduling interval determination module is used to determine the job scheduling interval according to the number of available GPUs in the cluster, the total number of GPUs requested by the job, and the job queue information; The job sorting module is used to update the job priority based on the job runtime information and sort the jobs by the maximum shared efficiency first method; The job determination module is used to traverse the jobs in the job queue and determine jobs to be run and jobs to be preempted; the jobs to be run are jobs that have sufficient resources to be placed after being sorted according to priority, and the jobs to be preempted are jobs that do not have sufficient resources to be placed after being sorted according to priority; The job location determination module is used to call the optimal packaging strategy and the job co-location interference prediction model to determine the placement location of the job to be run; The job placement module is used to place jobs and update cluster status; The monitoring and updating module is used to monitor the submission of new jobs and update the scheduling interval. If a new job is submitted, it jumps to the interference portrait analysis module; otherwise, it jumps to the scheduling interval determination module.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the GPU sharing scheduling method supporting heterogeneous job scales as described in any one of claims 1-6.

9. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the GPU sharing scheduling method supporting heterogeneous job scales described in any one of claims 1 to 6 is implemented.

Images