A job scheduling method and system for domestic heterogeneous computing clusters

Abstract

The present disclosure proposes a method and system for scheduling jobs for domestic heterogeneous computing power clusters. The method includes: in a scheduling cycle, according to the resource amount of the job to be scheduled, the computing power type request, and the computing power type label of the node, the candidate nodes are screened out from the heterogeneous computing power cluster; considering the performance differences of heterogeneous computing power resources, the weights of the candidate nodes are calculated based on weighted polling, and the candidate node with the highest weight is used as the first target node; the resource usage space of the candidate nodes is scored according to the proportion of various resources in the heterogeneous computing power cluster and the candidate nodes, and the candidate node with the highest score is used as the second target node; the final target node is randomly selected, and the job to be scheduled is scheduled to the final target node. The present disclosure improves the node pre-selection efficiency by adding a primary screening in the node pre-selection stage, and in the node optimization stage, the performance differences of heterogeneous computing power resources and the integrity of the cluster are considered, thereby overcoming the problems of load imbalance and incompatibility of heterogeneous computing power.

Description

Technical Field

[0001] The present invention relates to the field of cloud computing technology, and in particular to a job scheduling method and system for domestic heterogeneous computing power clusters. Background Art

[0002] With the rise of large language models, people's demand for computing power has reached an unprecedented level. AI computing power has also begun to show diversified and heterogeneous characteristics, and a variety of high-quality domestic AI chip computing power resources have emerged one after another.

[0003] In order to meet the growing application needs, many data centers currently deploy multiple large-scale heterogeneous computing resource clusters to form one or more physical resource pools for upper-level business use. The job scheduler schedules different jobs to run on appropriate nodes according to the pre-specified resource types and specifications and the logic specified by the scheduling strategy. Currently, the scheduling strategy is mainly implemented using Kubernetes (K8s for short) container orchestration and scheduling engine technology. Research on efficient scheduling strategies is conducive to improving the load balancing capabilities of the cluster and the utilization of computing resources. Existing scheduling strategies can be roughly divided into three categories: K8s default scheduling strategy, scheduling strategies improved based on K8s default scheduling strategy, and scheduling strategies provided by third-party scheduling frameworks.

[0004] The inventors found that the above scheduling strategy has the following problems:

[0005] First, the pre-selection algorithm of the K8s default scheduling strategy needs to compare the Pod's resource request with the amount of free resources of each node in the cluster one by one. For large heterogeneous computing clusters containing multiple AI chips, this node pre-selection algorithm is very inefficient. At the same time, the K8s default scheduling strategy lacks a load balancing mechanism. It only schedules based on the availability of node resources, which may cause some nodes to be overloaded while other nodes are relatively idle. Since the score of each candidate node is not determined based on the integrity of the cluster, the load of the computing cluster is unbalanced.

[0006] Second, the load balancing indicators in the scheduling strategy improved based on the K8s default scheduling strategy mainly refer to the CPU, GPU, and memory, and do not consider the performance indicators related to domestic AI chips. Therefore, these strategies are not applicable to heterogeneous computing clusters containing domestic AI chips.

[0007] Third, the load balancing algorithm in the scheduling strategy provided by the third-party scheduling framework does not take into account the performance differences between heterogeneous computing resources such as domestic AI chips. This leads to low utilization of computing resources of domestic AI chips, resulting in a waste of domestic AI computing resources. Summary of the invention

[0008] In order to solve the above-mentioned problems, the present invention proposes a job scheduling method and system for domestic heterogeneous computing power clusters. The proposed scheduling strategy improves the node pre-selection efficiency by adding initial screening in the node pre-selection stage; in the node optimization stage, two node optimization algorithms are set considering the performance differences of heterogeneous computing power resources and the integrity of the cluster, and the final target node is randomly determined based on two candidate preferred nodes, which specifically solves the problems of load imbalance and incompatibility of heterogeneous computing power.

[0009] In order to achieve the above object, the present invention adopts the following technical solution:

[0010] In a first aspect, the present invention provides a method for scheduling jobs for a domestic heterogeneous computing cluster, comprising:

[0011] In a scheduling cycle, candidate nodes are selected from the heterogeneous computing power cluster based on the resource amount, computing power type request, and computing power type label of the job to be scheduled;

[0012] Considering the performance differences of heterogeneous computing resources, the weight of each candidate node is calculated based on smooth weighted polling, and the candidate node with the highest weight is used as the first target node; the resource usage space of each candidate node is scored according to the proportion of various resources in the heterogeneous computing cluster and candidate nodes, and the candidate node with the highest score is used as the second target node; the first target node and the second target node are randomly probabilistically obtained to obtain the final target node;

[0013] Schedule the jobs to be scheduled to the final target node.

[0014] Preferably, the computing power type label of the node is: the nodes in the heterogeneous computing power cluster are classified in advance, and the computing power type label is set for the node.

[0015] Preferably, the specific steps of selecting candidate nodes from the heterogeneous computing power cluster according to the resource amount of the job to be scheduled, the computing power type request, and the computing power type label of the node are as follows:

[0016] Based on the computing power type label of the node, select the nodes with the same computing power type request as the job to be scheduled from the heterogeneous computing power cluster to obtain preliminary candidate nodes;

[0017] The resource amount of the job to be scheduled is compared with the available resources of the preliminary pre-selected nodes, and the nodes that meet the resource amount requirements of the job to be scheduled are screened out to obtain the candidate nodes.

[0018] Preferably, the performance differences of heterogeneous computing resources are taken into consideration, and the weight of each candidate node is calculated based on smooth weighted polling. The specific process is:

[0019] In a scheduling cycle, the current initial weight of the candidate node is the cumulative sum of the weight values ​​of the historical scheduling cycle; the weight value of the historical scheduling cycle is calculated based on the operation status of various resources in the candidate node in the historical scheduling cycle;

[0020] The weight of the candidate node is obtained by subtracting the sum of the weights of all candidate nodes from the current weight of the candidate node.

[0021] Preferably, the resource usage space of each candidate node is scored according to the proportion of various resources in the heterogeneous computing power cluster and the candidate node, and the specific process is as follows:

[0022] Calculate the occupancy ratio of various resources in the heterogeneous computing power cluster and normalize them to obtain the normalized weight coefficient vector;

[0023] Calculate the occupancy ratio of each idle resource in each candidate node in the heterogeneous computing power cluster, and obtain the weight vector of each idle resource in the node;

[0024] After performing the inner product operation on the normalized weight coefficient vector and the transpose of the weight vector of each idle resource in the node, the final score of the candidate node is obtained by subtracting it from 1.

[0025] Preferably, the step of scheduling the job to be scheduled to the final target node specifically includes: obtaining the final target node ID, and scheduling the job to be scheduled to be executed on the final target node according to the ID.

[0026] Preferably, Volcano is used as the job scheduling engine.

[0027] In a second aspect, the present invention provides a job scheduling system for a domestic heterogeneous computing power cluster, comprising:

[0028] The heterogeneous computing resource layer is configured to provide a variety of heterogeneous computing resources for model training at the user application layer;

[0029] The heterogeneous resource scheduling layer is configured to provide a scheduling strategy according to the job to be scheduled; it includes a node pre-selection module and a node optimization module; the node pre-selection module is used to select candidate nodes from the heterogeneous computing power cluster according to the resource amount, computing power type request, and computing power type label of the node of the job to be scheduled in a scheduling cycle; the node optimization module is used to consider the performance difference of heterogeneous computing power resources, calculate the weight of each candidate node based on smooth weighted polling, and take the candidate node with the highest weight as the first target node; score the resource usage space of each candidate node according to the proportion of various resources in the heterogeneous computing power cluster and the candidate nodes, and take the candidate node with the highest score as the second target node; and obtain the final target node by random probability of the first target node and the second target node;

[0030] The user application layer is configured to perform data upload, model training, job management and image management functions in a visual interface; the data upload function is used to receive the data set and code required for model training uploaded by the user; the model training function is used to create a model training job based on the data set and code required for model training; the job management function is used to start, stop and delete the created job based on the scheduling policy; the image management is used to provide image upload and deletion.

[0031] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a job scheduling method for a domestic heterogeneous computing cluster as described in the first aspect.

[0032] In a fourth aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of a job scheduling method for a domestic heterogeneous computing cluster as described in the first aspect are implemented.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] (1) The present invention designs a node pre-selection algorithm for fast tagging and searching based on multiple heterogeneous computing resources. By pre-classifying and labeling computing resource nodes, nodes that meet the job requirements can be quickly screened out according to the computing resource requirements of the job submitted by the user, shortening the node pre-selection time and achieving the purpose of improving the node pre-selection efficiency. This effectively solves the problem of low pre-selection efficiency of the K8s default scheduling strategy in large-scale heterogeneous computing clusters and improves the overall efficiency of job scheduling.

[0035] (2) The present invention designs a node optimization algorithm that performs weighted polling based on the performance differences of various heterogeneous computing resources. By fully considering the performance differences between various heterogeneous computing resources such as domestic AI chips, the weighted polling concept is applied to a heterogeneous computing cluster containing multiple AI chips. While completing the node optimization in the scheduling cycle, it can also ensure that the cluster load is more balanced, improve the utilization rate of domestic AI chip computing resources, and solve the problem of waste of domestic AI computing resources.

[0036] (3) The present invention also designs a node optimization algorithm that normalizes weights based on the integrity of multiple heterogeneous computing resource clusters. By normalizing the overall resource usage ratio of the heterogeneous computing cluster, the usage of each resource of each node is scored according to the overall performance of the cluster. While completing node optimization, it can also ensure that the load of the heterogeneous computing cluster is more balanced. This effectively solves the problem of unbalanced cluster load in the K8s default scheduling strategy.

[0037] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings constituting a part of the present disclosure are used to provide a further understanding of the present disclosure. The exemplary embodiments of the present disclosure and the description thereof are used to explain the present disclosure but do not constitute a limitation of the present disclosure.

[0039] Figure 1 The main flow chart of a method for scheduling jobs for a domestic heterogeneous computing cluster provided by the present disclosure;

[0040] Figure 2 A detailed flow chart of a method for scheduling jobs for a domestically produced heterogeneous computing cluster provided by the present disclosure;

[0041] Figure 3 The overall architecture diagram of a job scheduling system for domestic heterogeneous computing clusters provided in this disclosure. DETAILED DESCRIPTION

[0042] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0043] After market research, it was found that there is currently a lack of job scheduling systems in China that are suitable for heterogeneous computing clusters containing a variety of domestic and foreign AI chips. Most existing job scheduling systems only support Nvidia series GPU resource scheduling and lack the integration of a variety of domestic AI chip heterogeneous computing resources. At the same time, Nvidia's GPUs rely on a powerful software ecosystem, making most users more inclined to use GPUs for model training. However, the current threshold for using domestic AI chips is still too high to meet the current needs of efficient job scheduling. The localization of job scheduling systems for AI heterogeneous computing clusters is imminent.

[0044] Therefore, the present invention proposes a job scheduling method and system for heterogeneous computing power clusters, using Volcano as the job scheduling engine, and by designing a node pre-selection algorithm and two node optimization algorithms, it not only realizes the efficient scheduling of AI jobs in heterogeneous computing power clusters containing a variety of domestic AI chips, but also improves the utilization rate of domestic AI computing power resources. This solves the problem of domestic substitution of AI heterogeneous computing power job scheduling systems and the problem that the threshold for ordinary users to use domestic AI chip computing power resources for model training is too high.

[0045] Embodiment 1

[0046] like Figure 1 As shown, this embodiment discloses a job scheduling method for a heterogeneous computing power cluster, including the following steps:

[0047] S1: In a scheduling cycle, candidate nodes are selected from the heterogeneous computing power cluster according to the resource amount, computing power type request, and computing power type label of the job to be scheduled;

[0048] S2: Considering the performance differences of heterogeneous computing resources, the weight of each candidate node is calculated based on smooth weighted polling, and the candidate node with the highest weight is used as the first target node; the resource usage space of each candidate node is scored according to the proportion of various resources in the heterogeneous computing power cluster and candidate nodes, and the candidate node with the highest score is used as the second target node; the first target node and the second target node are randomly probabilistically obtained to obtain the final target node;

[0049] S3: Schedule the job to be scheduled to the final target node.

[0050] This embodiment provides a node pre-selection algorithm and two node optimization algorithms. The node pre-selection algorithm is a node pre-selection algorithm that performs fast tag search based on multiple heterogeneous computing resources, which can improve the efficiency of node pre-selection; the node optimization algorithm includes a node optimization algorithm that performs weighted polling based on the performance differences of multiple heterogeneous computing resources and a node optimization algorithm that performs weight normalization based on the integrity of multiple heterogeneous computing resource clusters. For different scheduling strategy requests, load balancing optimization is performed on node performance differences and cluster integrity.

[0051] like Figure 2 As shown, the method flow described in this embodiment will first initialize the heterogeneous computing power cluster (heterogeneous_cluster). Then, the job submitted by the user enters the queue of jobs to be scheduled, and then the jobs at the head of the queue are popped out in turn to execute the scheduling logic until the queue is empty. Next, the job to be scheduled enters the node pre-selection stage. In the node pre-selection stage, the candidate node list is obtained by calling the node pre-selection algorithm (NodePredictFn) for fast tag search based on multiple heterogeneous computing power resources described below. If the candidate node list is empty, the job will be put back into the queue and wait for subsequent scheduling; conversely, if the candidate node list is not empty, the node optimization stage is entered based on the candidate node list. In the node optimization stage, the first target node is obtained by calling the node optimization algorithm 1 (NodePriorityFn1) designed below; the second target node is obtained by calling the node optimization algorithm 2 (NodePriorityFn2) designed below; the first target node and the second target node are randomly probabilistically obtained The final target node, and finally the job is scheduled to be executed on the target node. The pseudo code implementation of the overall method is shown in Algorithm 1:

[0052]

[0053] (1) Node pre-selection algorithm

[0054] The basic principle of the algorithm is: in the pre-selection stage of the job scheduling cycle, each node in the heterogeneous computing power cluster is pre-labeled. Nodes with the same AI chip computing power resources are marked with the same label. The domestic and foreign AI chip computing power resources used in the present invention include: NVIDIA's GPU, Cambrian's MLU and Haiguang's DCU. In this way, when all nodes are screened for the first time, the nodes containing the heterogeneous computing power resources required by the user can be quickly screened out, achieving the effect of rapid pre-screening. Then, a secondary screening is performed based on the available resources of each node, and finally multiple candidate nodes are obtained for scoring in the node optimization stage.

[0055] The pseudo code of the algorithm is shown in Algorithm 2. First, obtain the AI ​​chip heterogeneous computing power resource type (request_accelerator_microchip_type) requested in the job (job) submitted by the user. Then, traverse each node in the heterogeneous computing power cluster and match it according to the pre-labeled label of each node. After the initial screening, a list of nodes (same_type_nodes) that meet the computing power resource type requested by the job is obtained. Then, a secondary screening is performed based on the node list, and each node in the list is traversed. If the available resources of the node meet the computing power resource request of the job, the node is a qualified candidate node. Finally, a set of all qualified candidate nodes (available_nodes) is returned.

[0056] By adding the initial screening pre-selection mechanism, the overall efficiency of node pre-selection can be greatly improved.

[0057]

[0058]

[0059] (2) A node selection algorithm based on weighted polling based on the performance differences of multiple heterogeneous computing resources

[0060] The node selection algorithm is mainly based on the improvement and optimization of the smooth weighted polling algorithm. Its basic principle is: first, set the initialization weight for each node in the cluster according to the performance of the various heterogeneous computing resources contained in each node. The calculation formula of the initialization weight is as follows:

[0061] W init =0.8(αCPU+βAI+ωVRAM)+0.1RAM+0.1STORAGE

[0062] Among them, CPU represents the number of CPU computing cores of the current node, AI represents the number of AI chips in the current node (AI can represent multiple types of AI chips such as GPU, MLU, DCU, etc.), VRAM represents the video memory of the AI ​​chip in the current node, RAM represents the total memory of the current node, and STORAGE represents the total storage of the current node. α, β, and ω can be customized weight coefficients according to the type of job submitted, and their sum always maintains α+β+ω=1. As the scheduling cycle progresses, the current weight of each node will be dynamically adjusted through the idea of ​​smooth weighted polling. The calculation formula is as follows:

[0063] W current [i] = W last [i]+W init [i]

[0064]

[0065] W target =Max(W current [1],W current [2],…,W current [n])-W total

[0066] Among them, W init [i] represents the initial weight of the i-th node, W current [i] represents the current weight of the i-th node, W last [i] represents the weight of the i-th node last time, W total Represents the sum of the initialization weights of all nodes, W target Indicates the weight of the target node after adjustment. The adjustment rule is: the current weight for the first time is equal to the initial weight, and the subsequent current weight is equal to the sum of the previous weight and the initial weight. Then, the node with the highest current weight is selected as the target node. In order to maintain the stability and fairness of the algorithm, the weight of the final target node is subtracted from the sum of the weights of all nodes. The pseudo code of the algorithm is shown in Algorithm 3.

[0067]

[0068] (3) Node selection algorithm based on weight normalization of the integrity of multiple heterogeneous computing resource clusters

[0069] The node selection algorithm is mainly to normalize the usage ratio of the overall resources of the heterogeneous computing power cluster, and score the usage of each resource of each node according to the overall performance of the cluster. While completing the node selection, it can also ensure that the load of the heterogeneous computing power cluster is more balanced. The specific steps are as follows:

[0070] The first step is to count and calculate the proportion α of each resource in the entire cluster. The proportion calculation formula is as follows:

[0071]

[0072] Among them, CPU total , RAM total , AI total , VRAM total , STORAGE total Respectively represent the total amount of CPU, memory, AI chip (including: NVIDIA GPU, Cambrian MLU, Haiguang DCU), video memory and storage resources in the entire heterogeneous computing cluster. avg , RAM avg , AI avg , VRAM avg , STORAGE avg They represent the average total occupancy of the CPU, memory, AI chip, video memory, and storage in the entire heterogeneous computing cluster. The average occupancy is a weighted moving average calculated by the historical occupancy and instantaneous occupancy of the resource. Taking the CPU as an example, the calculation formula is as follows:

[0073]

[0074] Among them, CPU t is the instantaneous CPU occupancy at time t, CPU t-1 is the instantaneous CPU occupancy at time (t-1), β is a smoothing factor (also called a decay factor), which usually takes a value between 0 and 1, indicating the weight of new data. A larger β means more emphasis on the latest data.

[0075] The second step is to normalize the proportion coefficients of the above resources to obtain the normalized proportion coefficient variables of each resource in the entire heterogeneous computing power cluster. First, the denominator Y used in weight normalization needs to be calculated. The calculation formula is as follows:

[0076]

[0077] Then, use the denominator Y calculated above to normalize the weight coefficient of each resource in the entire heterogeneous computing power cluster. The calculation formula is as follows:

[0078]

[0079] Then the normalized coefficients are combined into a normalized weight coefficient vector α, α is as follows:

[0080] α=[α CPU ,α RAM ,α AI ,αVRAM ,α STORAGE ]

[0081] The third step is to calculate the proportion vector A of each idle resource in each node of the heterogeneous computing power cluster. The calculation formula is as follows:

[0082]

[0083] Among them, R(CPU), R(RAM), R(AI), R(VRAM), and R(STORAGE) respectively represent the idle resources of CPU, memory, AI chip, video memory, and storage in each node; T(CPU), T(RAM), T(AI), T(VRAM), and T(STORAGE) respectively represent the total resources of CPU, memory, AI chip, video memory, and storage in each node.

[0084] The fourth step is to perform an inner product operation on the normalized weight coefficient vector α calculated in the previous article and the transpose of the weight vector A of each idle resource in the node, and then subtract 1 from the inner product result to obtain the final score of the node. The score calculation formula is as follows:

[0085] score = 1-αA T

[0086] In summary, the algorithm will use the occupancy weight of each resource in the entire cluster. If the occupancy weight coefficient of a resource is larger, it means that this resource has been used more, and the score of this node will be smaller. When selecting nodes, this node will not be considered as the target node. The pseudo code of this algorithm is shown in Algorithm 4.

[0087]

[0088]

[0089] Based on the existing scheduling strategy, this specific embodiment proposes a job scheduling method for domestic heterogeneous computing power clusters. The method uses Volcano as the job scheduling engine and applies the existing scheduling strategy to practice. The method not only realizes the efficient scheduling of AI jobs in heterogeneous computing power clusters containing multiple domestic AI chips, but also improves the utilization rate of domestic AI computing power resources. This solves the problem of domestic substitution of AI heterogeneous computing power job scheduling systems and the problem that the threshold for ordinary users to use domestic AI chip computing power resources for model training is too high.

[0090] Embodiment 2

[0091] This embodiment provides a job scheduling system for domestic heterogeneous computing power clusters. The overall architecture of the system is mainly divided into three layers from bottom to top, including the heterogeneous computing power resource layer, the heterogeneous resource scheduling layer, and the user application layer. The overall system architecture diagram is as follows: Figure 3 As shown, including:

[0092] The heterogeneous computing resource layer is configured to provide a variety of heterogeneous computing resources for model training at the user application layer;

[0093] The heterogeneous resource scheduling layer is configured to provide a scheduling strategy according to the job to be scheduled; it includes a node pre-selection module and a node optimization module; the node pre-selection module is used to select candidate nodes from the heterogeneous computing power cluster according to the resource amount, computing power type request, and computing power type label of the node of the job to be scheduled in a scheduling cycle; the node optimization module is used to consider the performance difference of heterogeneous computing power resources, calculate the weight of each candidate node based on smooth weighted polling, and take the candidate node with the highest weight as the first target node; score the resource usage space of each candidate node according to the proportion of various resources in the heterogeneous computing power cluster and the candidate nodes, and take the candidate node with the highest score as the second target node; and obtain the final target node by random probability of the first target node and the second target node;

[0094] The user application layer is configured to perform data upload, model training, job management and image management functions in a visual interface; the data upload function is used to receive the data set and code required for model training uploaded by the user; the model training function is used to create a model training job based on the data set and code required for model training; the job management function is used to start, stop and delete the created job based on the scheduling policy; the image management is used to provide image upload and deletion.

[0095] In a specific embodiment, the heterogeneous computing resource layer mainly provides users with a variety of heterogeneous computing resources for model training, which includes heterogeneous computing resource nodes containing a variety of AI chips, including CPU, NVIDIA GPU, Cambrian MLU and Haiguang DCU and other domestic and foreign AI chip computing resources.

[0096] In a specific embodiment, the heterogeneous resource scheduling layer mainly performs reasonable orchestration and optimal scheduling for jobs submitted by users. This layer is mainly based on K8s as the container orchestration and scheduling layer, and takes the scheduling strategy extension mechanism of the high-performance batch scheduling engine Volcano as an opportunity. By registering the node pre-selection algorithm and node optimization algorithm designed in the previous article as the node pre-selection algorithm plug-in NodePredictPlugin and the node optimization algorithm plug-in NodePriorityPlugin, the K8s default scheduling strategy is enhanced in the form of an algorithm plug-in. The execution logic of the algorithm plug-in enables the system to select the optimal target node according to the performance indicators and weights of the node, thereby realizing efficient scheduling of jobs.

[0097] In a specific embodiment, the user application layer mainly provides users with data upload, model training, job management and image management functions in a visual interface. The data upload function allows users to upload the data set and code required for model training. The model training function allows users to create model training jobs with one click. The job management function mainly facilitates users to start, stop and delete created jobs. Image management mainly provides users with image upload and deletion.

[0098] Embodiment 3

[0099] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of a job scheduling method for a domestic heterogeneous computing power cluster as described in the first embodiment above are implemented.

[0100] Embodiment 4

[0101] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of a job scheduling method for a domestic heterogeneous computing cluster as described in the first embodiment above are implemented.

[0102] The steps or modules involved in the above embodiments 2 to 4 correspond to those in embodiment 1. For the specific implementation, please refer to the relevant description of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0103] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A job scheduling method for domestic heterogeneous computing power clusters, characterized in that: include: In a scheduling cycle, candidate nodes are selected from the heterogeneous computing power cluster based on the resource amount, computing power type request, and computing power type label of the job to be scheduled; Considering the performance differences of heterogeneous computing resources, the weight of each candidate node is calculated based on smooth weighted polling, and the candidate node with the highest weight is selected as the first target node; The performance differences of heterogeneous computing resources are considered, and the weight of each candidate node is calculated based on smooth weighted polling. The specific process is: within a scheduling cycle, the current initial weight of the candidate node is the cumulative sum of the weight values ​​of the historical scheduling cycle; the weight value of the historical scheduling cycle is calculated based on the operation status of various resources in the candidate node within the historical scheduling cycle; The weight of the candidate node is obtained by subtracting the sum of the weights of all candidate nodes from the current weight of the candidate node; the resource usage space of each candidate node is scored according to the proportion of each type of resources in the heterogeneous computing power cluster and the candidate nodes, and the candidate node with the highest score is used as the second target node; the resource usage space of each candidate node is scored according to the proportion of each type of resources in the heterogeneous computing power cluster and the candidate nodes. The specific process is: calculating the occupancy ratio of each type of resource in the heterogeneous computing power cluster, and normalizing it to obtain a normalized weight coefficient vector; calculating the occupancy ratio of each idle resource in each candidate node in the heterogeneous computing power cluster, and obtaining the weight vector of each idle resource in the node; performing an inner product operation on the transpose of the normalized weight coefficient vector and the weight vector of each idle resource in the node, and subtracting it from 1 to obtain the final score of the candidate node; Perform random probability analysis on the first target node and the second target node to obtain the final target node; Schedule the jobs to be scheduled to the final target node.

2. A method for scheduling jobs for a domestic heterogeneous computing cluster as claimed in claim 1, characterized in that: The computing power type label of the node is: the nodes in the heterogeneous computing power cluster are classified in advance, and the computing power type label is set for the node.

3. A method for scheduling jobs for a domestic heterogeneous computing cluster as claimed in claim 2, characterized in that: The specific steps of selecting candidate nodes from the heterogeneous computing power cluster according to the resource amount of the job to be scheduled, the computing power type request, and the computing power type label of the node are as follows: Based on the computing power type label of the node, select the nodes with the same computing power type request as the job to be scheduled from the heterogeneous computing power cluster to obtain preliminary candidate nodes; The resource amount of the job to be scheduled is compared with the available resources of the preliminary pre-selected nodes, and the nodes that meet the resource amount requirements of the job to be scheduled are screened out to obtain the candidate nodes.

4. A method for scheduling jobs for a domestic heterogeneous computing cluster as claimed in claim 1, characterized in that: The step of scheduling the job to be scheduled to the final target node specifically includes: obtaining the final target node ID, and scheduling the job to be scheduled to be executed on the final target node according to the ID.

5. A method for scheduling jobs for a domestic heterogeneous computing cluster as claimed in claim 1, characterized in that: Volcano is used as the job scheduling engine.

6. A job scheduling system for domestic heterogeneous computing clusters, characterized in that: include: The heterogeneous computing resource layer is configured to provide a variety of heterogeneous computing resources for model training at the user application layer; The heterogeneous resource scheduling layer is configured to provide a scheduling strategy according to the job to be scheduled; it includes a node pre-selection module and a node optimization module; the node pre-selection module is used to select candidate nodes from the heterogeneous computing power cluster according to the resource amount, computing power type request, and computing power type label of the node of the job to be scheduled in a scheduling cycle; the node optimization module is used to consider the performance difference of heterogeneous computing power resources, calculate the weight of each candidate node based on smooth weighted polling, and take the candidate node with the highest weight as the first target node; the performance difference of heterogeneous computing power resources is considered, and the weight of each candidate node is calculated based on smooth weighted polling. The specific process is: within a scheduling cycle, the current initial weight of the candidate node is the cumulative sum of the weight values ​​of the historical scheduling cycle; the weight value of the historical scheduling cycle is calculated based on the operation status of various resources in the candidate node in the historical scheduling cycle; The weight of the candidate node is obtained by subtracting the sum of the weights of all candidate nodes from the current weight of the candidate node; the resource usage space of each candidate node is scored according to the proportion of each type of resources in the heterogeneous computing power cluster and the candidate nodes, and the candidate node with the highest score is used as the second target node; the resource usage space of each candidate node is scored according to the proportion of each type of resources in the heterogeneous computing power cluster and the candidate nodes. The specific process is: calculating the occupancy ratio of each type of resource in the heterogeneous computing power cluster, and normalizing it to obtain a normalized weight coefficient vector; calculating the occupancy ratio of each idle resource in each candidate node in the heterogeneous computing power cluster, and obtaining the weight vector of each idle resource in the node; performing an inner product operation on the transpose of the normalized weight coefficient vector and the weight vector of each idle resource in the node, and subtracting it from 1 to obtain the final score of the candidate node; Perform random probability analysis on the first target node and the second target node to obtain the final target node; The user application layer is configured to perform data uploading, model training, job management and image management functions in a visual interface; the data uploading function is used to receive the data set and code required for model training uploaded by the user; the model training function is used to create a model training job according to the data set and code required for model training; The job management function is used to start, stop and delete created jobs based on the scheduling policy; The image management is used to provide image uploading and deletion.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by the processor, the steps of a job scheduling method for a domestic heterogeneous computing power cluster as described in any one of claims 1 to 5 are implemented.

8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in a job scheduling method for a domestic heterogeneous computing cluster as described in any one of claims 1-5 are implemented.

Images