A deep learning training cluster scheduling method, device and computer-readable storage medium

Abstract

The present invention provides a deep learning training cluster scheduling method, device and computer-readable storage medium, the cluster scheduling method comprising: collecting data related to deep learning training tasks; constructing an effective multi-configuration deep learning training task duration estimation model and performing incremental training on the model; predicting the duration of tasks that arrive in the cluster successively to generate task durations under multiple configurations of deep learning tasks; combining periodic scheduling based on the prediction results with trigger-based scheduling results based on the task queue status to integrate comprehensive scheduling results. The deep learning training cluster scheduling method of the present invention can, on the one hand, effectively solve the defects of existing deep learning training cluster scheduling technologies, such as the difficulty in predicting task durations under multiple configurations, the scheduling scheme ignoring the CPU, and the difficulty in coping with continuously arriving tasks; on the other hand, it can achieve reasonable scheduling of deep learning training clusters, improve cluster resource utilization, optimize task execution efficiency, etc.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology and relates to a deep learning training cluster scheduling method, device and computer-readable storage medium. Background Art

[0002] In recent years, the application of deep learning in fields such as computer vision, natural language processing, speech, and recommender systems has rapidly increased. Deep learning training tasks consume significant resources, particularly computing resources such as GPUs and CPUs. As model complexity and data size continue to increase, the hardware requirements for these tasks are becoming increasingly demanding. In GPU clusters, a seemingly paradoxical phenomenon is often observed: while most GPUs are allocated to various tasks, their computing power is actually underutilized. This "high allocation, low utilization" situation results in a significant waste of valuable GPUs, while other tasks requiring GPU resources may be forced to wait due to a lack of allocation. Our research has found that a key factor contributing to this "high allocation, low utilization" situation is the inefficient use of CPU resources in existing scheduling systems. Deep learning training is a complex process that requires both GPU and CPU resources. The number of CPU cores allocated to deep learning training tasks directly affects the execution speed of these tasks.

[0003] The problems of existing deep learning training cluster scheduling methods mainly focus on the following two stages: configuration evaluation and scheduling decision.

[0004] Existing configuration evaluation methods for DLT task duration estimation rely on simple estimates of model structure and GPU performance, or use a large number of measured samples for estimation, but fail to consider the impact of the CPU on DLT tasks. Furthermore, faced with a vast number of configuration options, existing configuration evaluation methods cannot balance efficiency and accuracy.

[0005] Existing DLT scheduling solutions often rely on manually configured or pre-set fixed CPU allocation strategies. A few approaches adjust the number of CPUs based on task characteristics, but these methods fail to guarantee optimal CPU allocation at the cluster level. Furthermore, when cluster schedulers employ static scheduling, they are unable to dynamically adapt to environmental changes, potentially causing long-running tasks to block shorter ones.

[0006] Therefore, how to provide a deep learning training cluster scheduling solution to solve the defects of existing deep learning training cluster scheduling technology, such as the difficulty in predicting task duration under various configurations, the scheduling solution ignoring the CPU and the difficulty in coping with continuous arrival problems, has become a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the Invention

[0007] In view of the shortcomings of the prior art described above, the purpose of the present invention is to provide a deep learning training cluster scheduling method, system, device and computer-readable storage medium to address the shortcomings of the existing deep learning training cluster scheduling technology, such as the difficulty in predicting the task duration under various configurations, the scheduling scheme ignoring the CPU, and the difficulty in coping with continuously arriving tasks.

[0008] To achieve the above and other related objectives, the present invention provides a deep learning training cluster scheduling method, comprising:

[0009] S1, collects data related to deep learning training tasks, including task resource requirements, cluster resource status data, and task running status data.

[0010] S2, analyzes the relevant characteristics of task execution duration and resource utilization, builds an effective multi-configuration deep learning training task duration estimation model, and performs incremental training on it.

[0011] S3 predicts the duration of tasks that arrive in the cluster one after another to generate task durations under multiple configurations of deep learning tasks;

[0012] S4, combines the periodic scheduling based on the prediction results and the triggered scheduling results based on the task queue status to integrate the comprehensive scheduling results.

[0013] Another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the deep learning training cluster scheduling method when executed by a processor.

[0014] The last aspect of the present invention provides a deep learning training cluster scheduling device, including: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the deep learning training cluster scheduling device executes the deep learning training cluster scheduling method.

[0015] Beneficial effects:

[0016] The deep learning training cluster scheduling method, device, and computer-readable storage medium described in the present invention can, on the one hand, effectively address the shortcomings of existing deep learning training cluster scheduling technologies, such as the difficulty in predicting task durations under various configurations, the scheduling scheme ignoring the CPU, and the difficulty in coping with continuously arriving tasks; on the other hand, it can achieve reasonable scheduling of deep learning training clusters, improve cluster resource utilization, optimize task execution efficiency, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flow chart of the cluster scheduling method of the present invention;

[0018] Figure 2 Schematic diagram of the process of step S3 in the cluster scheduling method of the present invention;

[0019] Figure 3 Schematic diagram of the process of step S4 in the cluster scheduling method of the present invention;

[0020] Figure 4 Schematic diagram of a configuration optimization mechanism based on key resource bottlenecks in a cluster scheduling method according to an embodiment of the present invention;

[0021] Figure 5 The experimental results of the cluster scheduling method according to the embodiment of the present invention on the overall task completion time and average turnaround time;

[0022] Figure 6 FIG. 1 is a schematic diagram of the principle structure of a cluster scheduling system in one embodiment of the present invention. DETAILED DESCRIPTION

[0023] The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0024] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. Therefore, the illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0025] Example 1

[0026] A deep learning training cluster scheduling method includes the following steps: (e.g. Figure 1 )

[0027] S1, collects data related to deep learning training tasks, including task resource requirements, cluster resource status data, and task running status data.

[0028] In this example, we collected execution time data for various deep learning tasks in different configurations within a cluster, accumulating a large number of samples. Based on this experimental data, we calculated the optimal number of CPU cores for each task under different GPU configurations.

[0029] Specifically, the deep learning training task-related data includes:

[0030] The resource requirements of the task, including CPU resource requirements and GPU resource requirements;

[0031] Cluster resource status data, including CPU usage and utilization, and GPU usage and utilization on nodes in the cluster;

[0032] Task running status data, including task type, batch size, dataset, and running time.

[0033] S2, analyzes the relevant characteristics of task execution duration and resource utilization, and builds an effective multi-configuration deep learning training task duration estimation model.

[0034] The S2 includes:

[0035] S21, calculation of the optimal number of CPU cores.

[0036] The optimal number of CPU cores represents the number that achieves the optimal balance between performance and resources. Extensive field testing and theoretical derivation show that before reaching the optimal number of CPU cores, increasing the number of CPU cores leads to significant performance improvements, with the system's performance improvement exhibiting a logarithmic growth pattern. However, once this critical point is exceeded, further increases in the number of CPU cores lead to a linear performance improvement, with the improvement becoming minimal and sometimes even causing performance degradation. The optimal number of CPU cores corresponds to the critical point where the performance curve transitions from logarithmic to linear.

[0037] The calculation process is as follows: collect task completion time data under different CPU core numbers, divide the data into two parts before and after the critical point, fit the logarithmic function and linear function respectively, then calculate the sum of the errors of the two parts, and finally select the point with the smallest error as the optimal CPU core number.

[0038] Specifically, define the optimal number of CPU cores is such that the error function The smallest positive integer value, that is, the optimal number of CPU cores. is the error function, defined as:

[0039]

[0040]

[0041]

[0042] here, yes The number of data points, yes The number of data points, and is the total observed running time, is a logarithmic function: , is a linear function: . and This is a performance parameter obtained through curve fitting. By traversing the possible numbers of CPU cores, the corresponding error function is calculated, where the minimum value of the error function corresponds to the optimal number of CPU cores.

[0043] This approach not only finds the most cost-effective CPU configuration point, avoiding resource waste, but also ensures optimal system performance. It's important to note that this critical point varies depending on the specific application scenario, workload characteristics, and system architecture, so it needs to be determined based on actual data.

[0044] S22, estimator model training method.

[0045] The estimator includes two machine learning models: the optimal CPU core prediction model and the performance extrapolation model. The optimal CPU core prediction model is used to predict the optimal number of CPU cores, and the performance extrapolation model is used to predict the final time.

[0046] The optimal CPU core prediction model is implemented using a random forest, using mean squared error as the loss function. Input features are the inter-layer information of the deep learning model converted to numerical features using one-hot encoding, the model type, the GPU configuration, and the batch size. The training label is the optimal number of CPU cores obtained in step S21. The deep learning model is the network model used for the deep learning training task.

[0047] The performance extrapolation model includes five time prediction models and an integration model. It adopts a phased modeling approach. First, the five time prediction models are used to predict the time of the five key computing stages: data processing, forward propagation, backpropagation, parameter update, and data movement. Then, the integration model is used to integrate the predicted time of the five stages into a final time.

[0048] Among them, the time prediction models of the five stages use the same model structure and parameter settings. The only difference is that the training data selects the running time from different stages.

[0049] Specifically, the time prediction model was implemented using a random forest algorithm, with 2-fold cross-validation used for hyperparameter optimization to select the optimal parameter combination. Mean squared error was used as the loss function. Input features were the optimal number of CPU cores for the deep learning model, performance parameters, the baseline GPU configuration, the data processing runtime of the deep learning model using the baseline GPU configuration and the optimal number of CPU cores, the GPU configuration to be predicted, and the number of CPU cores to be predicted. Training labels were the computation runtime for the predicted GPU and CPU configurations.

[0050] The ensemble model was implemented using the random forest algorithm, with mean squared error as the loss function. Input features were the optimal number of CPU cores, the baseline GPU configuration, the five-stage and full runtime of the deep learning model using the baseline GPU configuration and the optimal number of CPU cores, the predicted GPU configuration and the predicted number of CPU cores, and the five-stage runtime of the deep learning model using the predicted GPU configuration and the predicted number of CPU cores. The training labels were the runtime of the deep learning model using the predicted GPU configuration and the predicted number of CPU cores.

[0051] Furthermore, the estimator includes a historical model knowledge base, which includes the deep learning model's feature vector, the model's optimal number of CPU cores, performance parameters, and estimated duration information. The feature vector includes inter-layer information of the deep learning model, converted into numerical features via one-hot encoding, the model type, GPU configuration, and batch size.

[0052] When a new deep learning training task arrives, the estimator determines whether there is a corresponding historical prediction model for the input deep learning training task through feature vector matching. If the match is successful, the system can directly obtain the optimal number of CPU cores and performance parameters stored persistently in the model, thereby avoiding repeated calculation overhead.

[0053] After the deep learning model task is completed, the model's optimal number of CPU cores, performance parameters, and estimated duration information will be written to the persistent storage of the historical model knowledge base to fill model gaps and modify original data.

[0054] This approach is designed to ensure that the system can continuously optimize its prediction accuracy as cluster resources dynamically change and task loads diversify. By systematically collecting and analyzing actual task execution time data, the estimator can be incrementally updated to continuously improve the model's prediction accuracy.

[0055] The specific implementation includes the following key steps:

[0056] Data Collection and Storage: After a task is completed, the system records information such as the actual execution time, the number of CPU cores used, and the GPU configuration, and stores this data in a database. By accumulating more real-world data, the system can obtain richer samples for modeling and analysis.

[0057] Iterative model training: The system initiates the model retraining process by setting specific trigger conditions (such as periodicity or data volume thresholds), using the latest data sets to update model parameters and achieve dynamic improvement in predictive capabilities.

[0058] Incremental online learning: During the adaptive optimization process, the estimator adopts an incremental learning strategy to dynamically update model parameters through the real-time access of new data, enabling the system to quickly adapt to new task characteristics and cluster environment changes, thereby maintaining the efficiency and accuracy of predictions.

[0059] S3 predicts the duration of tasks that arrive in the cluster one after another to generate task durations under multiple configurations of deep learning tasks;

[0060] The specific steps include: using the method in step 2 to build a multi-configuration deep learning training task duration estimation model, and training it based on the data related to the deep learning training task in step S1; analyzing the possible configuration schemes for the tasks that arrive in the cluster one after another; inputting the task configuration scheme into the duration estimation model, and combining the current cluster resource status data and task running status data to calculate the estimated duration of the task under the current configuration.

[0061] The estimation algorithm consists of the following stages:

[0062] Prediction of optimal CPU core count and performance parameters: For heterogeneous GPU configurations, the system performs quantitative analysis on each configuration and predicts the optimal CPU core count and performance parameters.

[0063] Evaluation of estimated samples under the baseline GPU configuration: Select a baseline GPU configuration scheme, predict the estimated samples under the optimal number of CPU cores, and obtain the execution time measurement under the combination of the baseline GPU configuration and the optimal number of CPU cores;

[0064] Performance generalization across GPU configurations: Based on the runtime of the baseline GPU configuration at the optimal number of CPU cores, the estimator's performance extrapolation model generalizes the runtime to other GPU configurations at their respective optimal number of CPU cores. Based on the runtime information at the optimal number of CPU cores for each GPU configuration, the runtime is estimated for different CPU core counts to generate a complete performance function.

[0065] like Figure 2A complete flowchart for multi-configuration performance prediction for a deep learning task. Specifically:

[0066] When a task is submitted to the cluster, the system collects basic task data, including key parameters such as batch size, model type, and dataset. Literature analysis shows that when a DLT's CPU configuration falls below the optimal number of cores, the relationship between the number of cores and processing efficiency is logarithmic. This is due to the significant performance gains from increased parallel processing capabilities. However, when the optimal number of cores is exceeded, the performance gain flattens out, exhibiting a linear pattern.

[0067] After obtaining the task features, all feasible configuration solutions C are generated and checked whether the task exists in the historical knowledge base.

[0068] like Figure 2 As shown in , the estimation model maintains a historical model knowledge base and determines whether there is a corresponding historical prediction model through feature vector matching.

[0069] If there is a matching record, the optimal number of CPU cores and performance parameter information stored under configuration C can be directly obtained to avoid repeated calculations;

[0070] If it is a new task, the benchmark execution time needs to be estimated through sample estimation. As an example, a performance model is constructed by executing 1% and 2% scale sampling tasks, and the task execution time is decomposed into a fixed initialization time and an iterative execution time that is linearly related to the scale. The sample estimation will obtain the time for data processing, forward propagation, backpropagation, parameter update, and data movement stages; then, a performance extrapolation model is used, which includes the five-stage time prediction model based on random forest mentioned above and an integrated model based on random forest. The optimal number of CPU cores, performance parameters and execution time obtained under the baseline GPU configuration are input to predict the expected execution time under other GPU configuration schemes.

[0071] Ultimately, all prediction results will be stored in the historical model knowledge base. Through the system's adaptive optimization mechanism, including data collection and storage, model iterative training, and incremental online learning, the prediction accuracy will be continuously improved, enabling the system to adapt to the dynamic changes in cluster resources and the diverse needs of task loads.

[0072] The following is an example of deep learning task estimation: Assume that two deep learning tasks A and B enter the system, and the inter-layer information, model type, and batch size of their deep learning models have been determined.

[0073] The optimal CPU core prediction model in S22 is used to obtain the optimal number of CPU cores and performance parameters under the baseline GPU configuration, as shown in the following table. The baseline GPU configuration is manually selected, a1, a2, b1, and b2 are the predicted curve fitting performance parameters, and 10 and 15 are the optimal CPU core numbers for tasks A and B under the baseline GPU configuration, respectively:

[0074]

[0075] We then evaluated the estimated sample under the baseline GPU configuration, obtaining the estimated runtime for the baseline GPU configuration and the optimal number of CPU cores. This is shown in the table below, where * indicates the optimal number of CPU cores and baseline GPU configuration.

[0076]

[0077] We perform cross-GPU performance generalization, using the performance extrapolation model in S22 to predict the completion time for other CPU numbers and GPU models, generating estimated durations for all feasible configurations for subsequent scheduling. The following table shows the number of CPUs and the GPU configuration type, where * indicates the optimal CPU core and baseline GPU configuration:

[0078]

[0079] S4, combines the periodic scheduling based on the prediction results and the triggered scheduling results based on the task queue status to integrate the comprehensive scheduling results. (e.g. Figure 3 )

[0080] Cluster-level scheduling is based on the task duration of a single deep learning task obtained in steps S1-S3.

[0081] Cluster-level scheduling includes: a locally optimal triggered scheduler and a globally optimal periodic scheduler. The locally optimal triggered scheduler is triggered when a task state transitions, while the globally optimal periodic scheduler is triggered periodically.

[0082] The details are as follows:

[0083] Locally optimal triggered scheduler: This scheduler uses the Shortest Time-to-Completion First (STCF) strategy to respond immediately to task state transitions and optimize resource allocation. Its execution process consists of two main phases:

[0084] (1) Resource configuration screening: constructing a feasible configuration space based on the characteristics of tasks in the waiting queue and the currently available computing resources;

[0085] (2) Configuration selection and task scheduling: Select the optimal configuration and deploy tasks based on the completion time minimization criterion in the feasible configuration space.

[0086] Globally optimal periodic scheduler: This scheduler is implemented based on a simulated annealing heuristic method to generate a scheduling plan for the next period.

[0087] The solution space for the global optimal scheduling solution is very large, and the time complexity of solving the scheduling solution by traversal is unacceptable. Experimental results show that even with small data sets, the computational overhead of the exhaustive scheduling strategy increases significantly. This problem is NP-hard.

[0088] To ensure the practicability of the scheduling algorithm, the present invention adopts a heuristic algorithm based on simulated annealing to achieve global optimal scheduling.

[0089] The specific implementation steps of the heuristic algorithm based on simulated annealing are as follows:

[0090] Initialization phase: Build the initial scheduling plan based on the Shortest Job First (SJF) strategy and set the initial system temperature parameters , cooling rate and running rounds.

[0091] In this embodiment, the initial temperature parameter is set The value is 1000, the cooling rate is 0.99, and the number of running rounds is 2000.

[0092] Solution Space Exploration:

[0093] Random sampling is performed in the neighborhood space of the current solution to generate candidate solutions. Specifically, while ensuring resource constraints, tasks in the existing scheduling scheme are randomly reallocated and optimized to fully explore the solution space.

[0094] Furthermore, in order to improve the efficiency of solution space exploration, a node information caching mechanism is introduced to reduce redundant calculations by caching the optimal solution server completion time during the iteration process.

[0095] During the iterative process of the simulated annealing algorithm, each round of iteration requires a performance evaluation of the existing solution and candidate solutions to quantify the optimization effect of the candidate solutions. This process requires repeated calculation of the task completion time of each solution. Due to the characteristics of the simulated annealing algorithm, the acceptance rate of the candidate solutions it generates is usually maintained at a low level. Experimental data shows that the acceptance ratio of candidate solutions does not exceed 40% of the total number of iterations, indicating that a specific solution may not be replaced until after multiple rounds of iterations. Considering that the candidate solutions generated in each round of iteration only fine-tune part of the location or configuration of the task, there is no need to recalculate the task execution time of all server nodes. Based on this observation, a server completion time caching mechanism for the optimal solution in the iterative process is implemented, which effectively reduces redundant calculations and significantly improves the execution efficiency of the algorithm.

[0096] Acceptance Criteria: For randomly generated new schedules, the system evaluates them based on the total completion time and the migration cost. The standard migration cost is half the cost of migrating all tasks.

[0097] The acceptance decision of the scheduling plan can be divided into the following four scenarios:

[0098] Scenario 1: Completion time is shortened and resource migration overhead is lower than the standard migration overhead: The system unconditionally accepts the scheduling plan;

[0099] Scenario 2. Completion time is shortened but resource migration overhead is higher than the standard migration overhead: The system accepts the scheduling plan with probability p1;

[0100] Scenario 3. Completion time is extended but resource migration overhead is lower than the standard migration overhead: The system accepts the scheduling plan with probability p2;

[0101] Scenario 4. Completion time is extended and resource migration overhead is higher than the standard migration overhead: The system accepts the scheduling plan with probability p3.

[0102] As an example, the probabilities p1, p2, and p3 are set to 0.75, 0.5, and 0.25, respectively. Specific instructions are as follows:

[0103] The probability values ​​are determined experimentally: under the same load and hardware environment, multiple sets of probability values ​​are used to randomly generate new scheduling schemes, and the overall completion time of the algorithm is measured.

[0104] The selected probability value combinations are shown in the following table, where each row is a set of values ​​for p1, p2, and p3:

[0105]

[0106] The logic for selecting multiple groups of probability values ​​is as follows: Since the overall completion time is the primary scheduling indicator and the impact of migration overhead is relatively small, the solutions corresponding to p1, p2, and p3 show a gradually deteriorating trend. Based on this, in this method, the values ​​of p1, p2, and p3 adopt a gradually decreasing arithmetic progression.

[0107] Different groups of probability values ​​were generated with a granularity of 0.05 and experiments were conducted. The experimental results show that when p1, p2, and p3 are set to 0.75, 0.5, and 0.25 respectively, the overall completion time of the algorithm is the shortest.

[0108] After obtaining the globally optimal scheduling solution, the scheduler compares the old and new solutions and performs task interruption and recovery operations. Thanks to Kubernetes (K8S) container live migration technology, the task interruption and recovery overhead is typically less than 1% of the total task execution time. For any deep learning training task X, the system will execute the corresponding strategy based on the following four scenarios:

[0109] Case 1: When the target node of the task assignment is consistent with the current running node, the system maintains the current execution state of the task and no migration operation is required.

[0110] Case 2: When the target node is inconsistent with the current running node, the system first performs a task interrupt operation on the source node, then migrates the task state to the target node and resumes execution.

[0111] Case 3: For a task in the running state, if the new solution puts it into the waiting state, the system will execute the task interrupt operation at the source node and persist its running state.

[0112] Case 4: For a task in the waiting state, if the new solution sets it to the running state, the system will initialize and start the task on the specified target node.

[0113] Final solution optimization:

[0114] The simulated annealing algorithm, as a heuristic search algorithm, introduces random perturbations to avoid local optimality traps, thereby gradually converging to the global optimal solution. However, due to the random nature of resource allocation, server resources may not be fully utilized when the algorithm terminates. Specifically, some servers may have idle computing resources during specific time periods, resulting in reduced cluster resource utilization. To optimize resource utilization efficiency, a configuration optimization mechanism based on critical path analysis was designed to further improve server resource utilization.

[0115] The schematic diagram of the configuration optimization mechanism based on key resource bottlenecks is as follows: Figure 4 .

[0116] Here are the steps:

[0117] Step (1) Identify the task X with the latest completion time in the current scheduling scheme. As can be seen from the figure, there are four tasks (A, B, C, D) executing in parallel. Task D has the latest completion time, which is task X with the latest completion time.

[0118] Step (2) Analyze the set of tasks that are executed in parallel with task X and calculate the maximum idle capacity of resources during the execution cycle of X. The figure shows that there is a parallel execution relationship between task D and task C, and the maximum idle capacity of system resources reaches 40%.

[0119] Step (3) aggregates the resource requirements of task X and the maximum available resource capacity. Based on this resource constraint, the optimal configuration with the shortest execution time is selected for X (the selection method is based on existing techniques). In this case, we can simply assume that 70% of the resources are allocated to task D.

[0120] Step (4) Select the adjacent tasks that were completed before X in the original scheduling plan and iterate through steps (1) to (3) until all tasks are optimized. That is, the optimization is performed in the order of C, B, and A. For tasks C and A, the maximum idle capacity is 0% and cannot be optimized. For task B, which runs in parallel with A and C, the maximum idle capacity is 10%, and 40% of the resources can be allocated to task B.

[0121] On the one hand, the cluster scheduling method described in this embodiment can effectively solve the defects of existing deep learning training cluster scheduling technology, such as the difficulty in predicting task duration under various configurations, the scheduling scheme ignoring the CPU, and the difficulty in coping with continuously arriving tasks; on the other hand, it can realize the reasonable scheduling of deep learning training clusters, improve cluster resource utilization, optimize task execution efficiency, etc.

[0122] To validate the effectiveness of the present invention, experiments compared it with the following methods: FCFS (First-Come, First-Served) is a basic scheduling method. This algorithm processes tasks based on their arrival order, without considering other task attributes. When a new task arrives, the system appends it to the end of the queue. The resource allocation for a task is determined manually. FCFS* (First-Come, First-Served) builds on the FCFS architecture and adopts the optimal resource allocation for the current task. Specifically, for a selected GPU configuration, the corresponding OCCC is selected. The SJF (Shortest Job First) algorithm prioritizes scheduling based on the optimal resource allocation. This method evaluates all pending tasks and selects the one with the shortest estimated completion time. The First Fit (FF) algorithm is a greedy resource scheduling mechanism. This algorithm traverses tasks in order of completion time, selecting the first configuration that meets resource requirements and has the shortest completion time for allocation, without considering potential subsequent optimal allocations. When a new task request arrives, the system scans all configurations starting from the starting position. The resource allocation process adopts an immediate decision-making mode, and the allocation operation is executed immediately once the resources that meet the constraints are identified.

[0123] The workload in the experiment uses task tracking data from Microsoft's production cluster to systematically evaluate 5,000 continuously arriving deep learning training tasks (DLT).

[0124] Experiments show that under this load, compared with the baseline method, the GPU utilization rate increased by an average of 56.54%, as shown in the following table.

[0125]

[0126] Comparison of overall task completion time (Makespan) and mean turnaround time (MTT) Figure 5 As shown in Figure 2 (where CoreTuner is the method of the present invention), the method of the present invention achieved an average improvement of 51.3% and 75.3% respectively.

[0127] Example 2

[0128] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, Figure 1 The cluster scheduling method.

[0129] At any possible level of technical detail combination, the present application may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present application.

[0130] A computer-readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. The computer-readable storage medium used herein is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0131] The computer-readable program described herein can be downloaded to each computing / processing device from a computer-readable storage medium, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device. The computer program instructions for performing the operation of this application can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and procedural programming languages ​​such as "C" language or similar programming languages. Computer readable program instructions can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partly on the user's computer, partly on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., utilizing an Internet service provider to connect through the Internet). In certain embodiments, personalized electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are customized by utilizing the state information of computer readable program instructions, which electronic circuits can execute computer readable program instructions, thereby realizing various aspects of the present application.

[0132] Example 3

[0133] This embodiment provides a cluster scheduling system, including: (such as Figure 6 )

[0134] Data collection module 41, used to collect data related to deep learning training tasks, including task resource requirements, cluster resource status data, and task running status data;

[0135] The multi-configuration duration estimation module 42 analyzes the characteristics of task execution duration and resource utilization, builds an effective multi-configuration deep learning training task duration estimation model, and performs incremental training on it to predict the duration of tasks that arrive in the cluster one after another, thereby generating task durations under multiple configurations of deep learning tasks;

[0136] The cluster-level optimal scheduling module 46 is used to combine the results of the periodic scheduling submodule based on the prediction results and the trigger scheduling submodule based on the task queue status to integrate the comprehensive scheduling results.

[0137] The cluster scheduling system described in this embodiment will be described in detail below with reference to the diagram. Figure 6 , which is a schematic diagram showing the principle structure of a cluster scheduling system in one embodiment. Figure 6 As shown, the cluster scheduling system 4 includes a data collection module 41, a multi-configuration duration estimation module 42, and a cluster-level optimal scheduling module 46. The multi-configuration duration estimation module 42 includes a historical model library submodule 43, an optimal CPU core prediction submodule 44, and a performance extrapolation submodule 45; the cluster-level optimal scheduling module 46 includes a periodic scheduling submodule 47 and a triggered scheduling submodule 48.

[0138] The data collection module 41 is used to collect data related to deep learning training tasks, including task resource requirements, cluster resource status data, and task running status data;

[0139] In this embodiment, data related to deep learning training tasks is collected, including task resource requirements, cluster resource status data, and task execution status data. In practice, execution time data for various deep learning tasks under different configurations can be collected within the cluster. This accumulated data can then be used to calculate the optimal number of CPU cores for each task under different GPU configurations.

[0140] The multi-configuration duration estimation module 42 analyzes the relevant characteristics of task execution duration and resource utilization, constructs an effective multi-configuration deep learning training task duration estimation model, and performs incremental training on it. This model is used to predict the duration of tasks arriving in the cluster, generating task durations for multiple deep learning task configurations. Specifically, it calculates the optimal number of CPU cores: The optimal number of CPU cores is the key to achieving the optimal balance between performance and resources, marking the critical point where the performance curve transitions from logarithmic to linear. Before this point, increasing the number of CPU cores results in a logarithmic increase in performance; beyond this point, the performance improvement becomes linear and the magnitude decreases, or even decreases. Task completion time data for different numbers of CPU cores is collected and divided into two parts, before and after the critical point. Logarithmic and linear functions are fitted to each part, and the sum of the errors in the two parts is calculated. The point with the smallest error is selected as the optimal number of CPU cores. Furthermore, the historical model library submodule 43 records information about tasks that have already run in the system, improving information reuse and overall system efficiency. The estimator maintains a historical model knowledge base and determines whether the input task has a corresponding historical prediction model through feature vector matching. If so, it directly obtains relevant information to avoid repeated calculations.

[0141] The optimal CPU core prediction submodule 44 analyzes task characteristics and the GPU environment to predict the optimal number of CPU cores. The performance extrapolation submodule 45 analyzes the optimal number of CPU cores and sample estimated durations obtained in the optimal CPU core prediction submodule 44 to predict the estimated durations for all configurations. The core phase of the estimation algorithm: Optimal CPU core number and performance parameter prediction: For heterogeneous GPU configurations, each configuration is quantitatively analyzed to predict its corresponding optimal number of CPU cores and performance parameters. Estimated sample evaluation under a baseline GPU configuration: A baseline GPU configuration is selected and the estimated sample under the optimal number of CPU cores is predicted to obtain execution time measurements for the baseline GPU configuration and the optimal number of CPU cores parameter combination. Cross-GPU configuration performance generalization: Based on the runtime obtained for the baseline GPU configuration under the optimal number of CPU cores, the performance generalization model is used to generalize the runtime to other GPU configurations under their respective optimal number of CPU cores. For each GPU configuration, based on the execution time information under the optimal number of CPU cores, the runtime under different numbers of CPU cores is estimated to generate a complete performance function.

[0142] The cluster-level optimal scheduling module 46 is used to combine the results of the periodic scheduling submodule based on the prediction results and the trigger scheduling submodule based on the task queue status to integrate the comprehensive scheduling results.

[0143] In this embodiment, the cluster-level optimal scheduling module 46 includes a periodic scheduling submodule 47 and a trigger scheduling submodule 48. Figure 3 The process shown shows the overall scheduling strategy. In this embodiment, the periodic scheduling submodule 47 adopts a simulated annealing algorithm to achieve global optimal scheduling, escapes from the local optimal solution by introducing a probabilistic acceptance mechanism, and globally optimizes in the solution space. The steps are as follows: Initialization phase: construct an initial scheduling scheme based on the shortest task first (SJF) strategy, and set the system initial temperature parameter T; Solution space exploration: randomly sample the neighborhood space of the current solution to generate candidate solutions, and randomly redistribute and optimize the configuration of tasks in the existing scheduling scheme while ensuring resource constraints; Acceptance criteria: If the performance index of the new scheduling scheme is better than the current scheduling scheme, the system accepts it; otherwise, accept the suboptimal scheduling according to the probability function to avoid the local optimal solution, and evaluate the randomly generated new scheduling scheme based on the overall completion time and migration cost. The acceptance judgment is divided into the following four scenarios:

[0144] Completion time is reduced and resource migration cost is lower than standard migration cost: Accept unconditionally.

[0145] Completion time is shortened but resource migration overhead is higher than standard migration overhead: Accept with probability p1.

[0146] The completion time is extended but the resource migration cost is lower than the standard migration cost: accept with probability p2.

[0147] Completion time is extended and resource migration overhead is higher than standard migration overhead: Accept with probability p3.

[0148] Temperature Control Mechanism: A dynamic cooling strategy is employed to gradually lower system temperature parameters, reducing the probability of accepting suboptimal solutions and promoting algorithm convergence to the global optimal solution. After obtaining the globally optimal scheduling solution, the new and old solutions are compared, and task interruption and recovery operations are performed. Leveraging Kubernetes (K8S) container live migration technology, the task interruption and recovery overhead is typically less than 1% of the total task execution time. Different strategies are implemented based on the target node for task assignment and the current running node.

[0149] In this embodiment, the triggered scheduling submodule 48 employs a shortest completion time first (STCF) strategy to instantly respond to task state transitions and optimize resource allocation. The execution process includes: Resource configuration screening: Building a feasible configuration space based on the characteristics of tasks in the waiting queue and currently available computing resources. Configuration selection and task scheduling: Selecting the optimal configuration within the feasible configuration space based on the minimum completion time criterion and then deploying the task.

[0150] It should be understood that the division of the various modules in the above system is merely a division of logical functions. In actual implementation, they may be fully or partially integrated into a single physical entity or physically separated. Furthermore, these modules may be implemented entirely in software called by a processing element, entirely in hardware, or partially in software called by a processing element, while others may be implemented in hardware. For example, module x may be a separate processing element or integrated into a single chip in the above system. Furthermore, module x may be stored in the form of program code in the memory of the above system, called by a processing element in the system to perform the functions of module x. The implementation of other modules is similar. These modules may be fully or partially integrated or implemented independently. The processing element described herein may be an integrated circuit with signal processing capabilities. During implementation, the steps of the above method or the above modules may be performed by hardware integrated logic circuits in the processor element or by software instructions. These modules can be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs). When a module is implemented by scheduling program code on a processing element, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor capable of calling program code. These modules can be integrated together to implement a system-on-a-chip (SOC).

[0151] The cluster scheduling system described in this embodiment, based on the cluster scheduling method, provides a specific cluster schedule at each moment and dynamically schedules cluster resources and load. Furthermore, the cluster schedule generated by the cluster scheduling system is integrated with the cluster manager to provide a reliable and beneficial scheduling decision basis for cluster management operations, thereby achieving efficient utilization of cluster resources.

[0152] Example 4

[0153] This embodiment provides a cluster scheduling device, including: a processor, a memory, a transceiver, a communication interface and / or a system bus; the memory and the communication interface are connected to the processor and the transceiver via the system bus and communicate with each other; the memory is used to store computer programs, the communication interface is used to communicate with other devices, and the processor and the transceiver are used to run the computer program, so that the cluster scheduling device executes each step of the cluster scheduling method described in Example 1.

[0154] The system bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. This system bus can be divided into an address bus, a data bus, a control bus, and so on. For ease of illustration, the diagram uses only a single thick line, but this does not imply a single bus or type of bus. The communication interface enables communication between the database access device and other devices (such as clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk drive.

[0155] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0156] In summary, the cluster scheduling method, system, device and computer-readable storage medium described in the present invention can, on the one hand, effectively solve the defects of existing deep learning training cluster scheduling technology, such as the difficulty in predicting task duration under various configurations, the scheduling scheme ignoring the CPU and the difficulty in coping with continuously arriving tasks; on the other hand, it can realize the reasonable scheduling of deep learning training clusters, improve cluster resource utilization, optimize task execution efficiency, etc.

[0157] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical principles disclosed herein are intended to be covered by the claims of the present invention.

Claims

1. A deep learning training cluster scheduling method, characterized in that: The following steps are involved: S1, collects data related to deep learning training tasks, including task resource requirements, cluster resource status data, and task running status data; S2, analyzes the relevant characteristics of task execution duration and resource utilization, and builds an effective multi-configuration deep learning training task duration estimation model; S3 predicts the duration of tasks that arrive in the cluster one after another to generate task durations under multiple configurations of deep learning tasks; S4, combines the periodic scheduling based on the prediction results with the triggered scheduling results based on the task queue status to integrate the comprehensive scheduling results; Step S2 includes: S21, calculation of the optimal number of CPU cores; The optimal number of CPU cores represents the number of CPU cores that can achieve the best balance between performance and resources; The calculation process is as follows: Task completion time data for different CPU core counts is collected, divided into two parts before and after the critical point, and fitted with a logarithmic function and a linear function respectively. The sum of the errors in the two parts is then calculated, and the point with the smallest error is finally selected as the optimal number of CPU cores. S22, estimator model training method; The estimator includes two machine learning models: an optimal CPU core prediction model and a performance extrapolation model. The optimal CPU core prediction model is used to predict the optimal number of CPU cores, and the performance extrapolation model is used to predict the final time. The optimal CPU core prediction model is implemented based on random forests, using mean squared error as the loss function. The input features are the inter-layer information of the deep learning model converted into numerical features through one-hot encoding, the model type, the GPU configuration, and the batch size. The training label is the optimal number of CPU cores obtained in step S21. The deep learning model is the network model used for the deep learning training task. The performance extrapolation model consists of five time prediction models and one integration model. It adopts a staged modeling approach. First, the five time prediction models are used to predict the time of the five key computing stages: data processing, forward propagation, backpropagation, parameter update, and data movement. Then, the integration model is used to integrate the predicted time of the five stages into a final time. The time prediction models for the five stages use the same model structure and parameter settings. The only difference is that the training data are selected from the running time of different stages. Specifically, the time prediction model is implemented using the random forest algorithm, and 2-fold cross-validation is used for hyperparameter optimization to select the optimal parameter combination. The mean squared error is used as the loss function. The input features are the optimal number of CPU cores of the deep learning model, performance parameters, baseline GPU configuration, the running time of the data processing phase of the deep learning model under the baseline GPU configuration and the optimal number of CPU cores, the GPU configuration to be predicted, and the number of CPU cores to be predicted. The training labels are the running time of the calculation phase under the predicted GPU configuration and CPU configuration. The integrated model is implemented using the random forest algorithm, with mean square error as the loss function. The input features are the optimal number of CPU cores, the baseline GPU configuration, the five-stage time and complete running time of the deep learning model under the baseline GPU configuration and the optimal number of CPU cores, the GPU configuration to be predicted and the number of CPU cores to be predicted, and the five-stage time of the deep learning model under the GPU configuration to be predicted and the number of CPU cores to be predicted. The training label is the running time of the deep learning model under the GPU configuration to be predicted and the number of CPU cores to be predicted.

2. A deep learning training cluster scheduling method according to claim 1, characterized in that: In step S1, the deep learning training task related data includes: The resource requirements of the task, including CPU resource requirements and GPU resource requirements; Cluster resource status data, including CPU usage and utilization, and GPU usage and utilization on nodes in the cluster; Task running status data, including task type, batch size, dataset, and running time.

3. A deep learning training cluster scheduling method according to claim 1, characterized in that: The estimator also includes a historical model knowledge base, which includes: feature vectors of deep learning models, the optimal number of CPU cores of deep learning models, performance parameters and estimated duration information; the feature vectors include inter-layer information of the deep learning models converted into numerical features through one-hot encoding, model type, GPU configuration and batch size; When a new deep learning training task arrives, the estimator determines whether there is a corresponding historical prediction model for the input deep learning training task through feature vector matching. If the match is successful, the system can directly obtain the optimal number of CPU cores and performance parameters stored persistently in the deep learning model, thus avoiding repeated calculation overhead; After the deep learning model task is completed, the optimal number of CPU cores, performance parameters, and estimated duration of the deep learning model will be written into the persistent storage of the historical model knowledge base to fill model gaps and modify original data. By systematically collecting and analyzing actual task execution time data, the estimator is incrementally updated to continuously improve the model's prediction accuracy.

4. A deep learning training cluster scheduling method according to claim 1, characterized in that: Step S3 includes: building a multi-configuration deep learning training task duration estimation model, analyzing possible configuration schemes for tasks arriving in the cluster based on the deep learning training task-related data in step S1; inputting the task configuration scheme into the duration estimation model, and combining the current cluster resource status data and task running status data to calculate the estimated duration of the task under the current configuration; The estimation algorithm consists of three core stages: Prediction of optimal CPU core count and performance parameters: For heterogeneous GPU configurations, the system performs quantitative analysis on each configuration and predicts the optimal CPU core count and performance parameters. Evaluation of estimated samples under the baseline GPU configuration: Select a baseline GPU configuration scheme, predict the estimated samples under the optimal number of CPU cores, and obtain the execution time measurement under the combination of the baseline GPU configuration and the optimal number of CPU cores; Performance generalization across GPU configurations: Based on the runtime obtained under the optimal number of CPU cores for the baseline GPU configuration, the performance generalization model is generalized to the execution time of other GPU configurations under the constraints of their respective optimal number of CPU cores. Within each GPU solution, the runtime information under the optimal number of CPU cores is used to estimate the runtime under different CPU core numbers to generate a complete performance function.

5. A deep learning training cluster scheduling method according to claim 4, characterized in that: The specific process of step S3 is: First, when a task is submitted to the cluster, the system collects basic feature data of the task, including batch size, model type, and dataset; After obtaining the task features, all feasible configuration solutions C are generated and checked whether the task exists in the historical knowledge base; The estimation model maintains a historical model knowledge base and determines whether there is a corresponding historical prediction model through feature vector matching; If there is a matching record, the optimal number of CPU cores and performance parameter information stored under configuration plan C is obtained to avoid repeated calculations; If it is a new task, the benchmark execution time is estimated through sample estimation. By executing sampled tasks, a performance model is constructed, and the task execution time is decomposed into a fixed initialization time and an iterative execution time that is linearly related to the scale. The sample estimation will obtain the time for data processing, forward propagation, backpropagation, parameter update, and data movement stages; then, using the performance extrapolation model, the optimal number of CPU cores, performance parameters and execution time obtained under the baseline GPU configuration are input to predict the expected execution time under other GPU configuration schemes.

6. A deep learning training cluster scheduling method according to claim 1, characterized in that: In step S4, two schedulers are included: a locally optimal triggered scheduler and a globally optimal periodic scheduler; wherein, the locally optimal triggered scheduler is triggered when the task state changes, and the globally optimal periodic scheduler is triggered periodically; The details are as follows: Locally optimal triggered scheduler: The locally optimal triggered scheduler uses the shortest completion time first (STCF) strategy to respond immediately to task state transitions and optimize resource allocation. Its execution process includes the following stages: (1) Resource configuration screening: constructing a feasible configuration space based on the characteristics of tasks in the waiting queue and the currently available computing resources; (2) Configuration selection and task scheduling: Select the optimal configuration and deploy tasks based on the completion time minimization criterion in the feasible configuration space; Globally optimal periodic scheduler: The globally optimal periodic scheduler is implemented based on the simulated annealing heuristic method to generate the scheduling plan for the next period.

7. A deep learning training cluster scheduling method according to claim 6, characterized in that: The globally optimal periodic scheduler is implemented based on a heuristic method of simulated annealing and is used to generate a scheduling plan for the next period; The specific implementation steps are as follows: Initialization phase: Build the initial scheduling plan based on the shortest task priority strategy and set the initial temperature parameters of the system , cooling rate and running rounds; Solution space exploration: Random sampling is performed in the neighborhood space of the current solution to generate candidate solutions. To improve the efficiency of solution space exploration, a node information caching mechanism is introduced to cache the optimal solution server completion time during the iteration process, reducing redundant calculations. Acceptance criteria: For randomly generated new scheduling solutions, the system evaluates them based on the overall completion time and migration cost; The standard migration cost is half of the cost of migrating all tasks; The acceptance of the scheduling plan is divided into the following four scenarios: Scenario 1: The completion time is shortened and the resource migration cost is lower than the standard migration cost: The system unconditionally accepts the scheduling plan; Scenario 2: The completion time is shortened but the resource migration overhead is higher than the standard migration overhead: the system accepts the scheduling plan with probability p1; Scenario 3: The completion time is extended but the resource migration cost is lower than the standard migration cost: the system accepts the scheduling plan with probability p2; Scenario 4: The completion time is extended and the resource migration overhead is higher than the standard migration overhead: the system accepts the scheduling plan with probability p3; After obtaining the globally optimal scheduling solution, the scheduler compares the old and new solutions and performs task interruption and recovery operations. For any deep learning training task X, the scheduler executes the corresponding strategy according to the following four situations: Case 1: When the target node of the task assignment is the same as the current running node, the system maintains the current execution status of the task and no migration operation is required; Case 2: When the target node is inconsistent with the current running node, the system first performs a task interrupt operation on the source node, then migrates the task state to the target node and resumes execution; Case 3: For a task in the running state, if the new solution sets it to the waiting state, the system will perform task interruption operation at the source node and persist its running state; Case 4: For a task in the waiting state, if the new solution sets it to the running state, the system will initialize and start the task on the specified target node; Final solution optimization: To optimize resource utilization efficiency, a configuration optimization mechanism based on critical path analysis was designed to further improve server resource utilization. The steps are as follows: Step (1) Identify the task X with the latest completion time in the current scheduling scheme; Step (2) Analyze the set of tasks that are executed in parallel with task X and calculate the maximum idle capacity of resources during the execution period of X; Step (3) aggregate the resource requirements of task X and the maximum free resource capacity, and based on this resource constraint, select the optimal configuration solution with the shortest execution time for X; Step (4) Select the adjacent tasks that are completed before X in the original scheduling plan, and iteratively execute steps (1) to (3) until all tasks are optimized.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it implements the deep learning training cluster scheduling method described in any one of claims 1 to 7.

9. A deep learning training cluster scheduling device, characterized in that: include: processor and memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the deep learning training cluster scheduling device performs the deep learning training cluster scheduling method according to any one of claims 1 to 7.

Images