Distributed learning task scheduling method and system for heterogeneous computing clusters

Abstract

The present invention discloses a distributed learning task scheduling method and system for heterogeneous computing clusters. In the scheme: for distributed learning training tasks, a scheduling objective function that meets the characteristics of the distributed learning task is designed, which comprehensively considers the computing performance, communication overhead and resource utilization of the scheduling scheme to achieve balanced scheduling between different distributed learning task characteristics. At the same time, when scheduling across nodes, an appropriate scheduling strategy is selected based on the corresponding solution search space size, and the search overhead of the allocation scheme and the specific score of the allocation scheme in the scheduling process are comprehensively considered. Ultimately, the best scheduling scheme is obtained with a smaller search overhead while meeting the characteristics of the distributed learning task, thereby achieving the effect of accelerating the distributed learning task.

Description

Technical Field

[0001] This invention relates to the field of operating system task scheduling technology, and in particular to a distributed learning task scheduling method and system for heterogeneous computing clusters. Background Technology

[0002] Faced with increasingly complex computing tasks, distributed learning is considered an inevitable development trend and has gradually become a research hotspot in the industry. In the iterative computation process of distributed learning tasks, in addition to the computing nodes themselves needing to train the task, different computing nodes also need to frequently synchronize training model parameters to ensure that the model can traverse the complete dataset, thereby guaranteeing that the final model is consistent with the model trained on a single machine. For distributed learning systems, communication and computational overhead exhibit an exponential growth trend with the increase in system scale. Furthermore, as the number of computing nodes increases, the computing cluster exhibits significant heterogeneity in performance and communication methods. The performance of devices within different computing nodes varies, and the communication methods between computing nodes and between different devices within a node also differ, which greatly increases the difficulty and importance of distributed learning task scheduling.

[0003] Therefore, distributed learning tasks require an efficient task scheduler to coordinate resources and workloads within the cluster, ensuring the execution efficiency of distributed learning tasks and the resource utilization of the computing cluster. Device performance is a significant factor affecting computational overhead; using computing devices with different performance levels as training platforms for distributed learning tasks will greatly impact the completion time. Furthermore, considering the high communication requirements of distributed learning tasks, the differences in communication overhead between computing resources caused by different task scheduling schemes will also significantly affect the completion time.

[0004] In summary, the scheduling of distributed learning tasks requires comprehensive consideration of two major influencing factors: the computing performance of the device and the communication overhead generated by the scheduling scheme. This has significant research value and practical significance for further improving the training speed of distributed learning tasks. Summary of the Invention

[0005] The purpose of this invention is to provide a distributed learning task scheduling method and system for heterogeneous computing clusters. It analyzes the specific computing resource requirements of training tasks and combines the communication overhead weights within the cluster with the performance information of computing devices to make more reasonable and efficient task scheduling, ensuring high computing performance and low communication overhead of training tasks, thereby improving the training speed of distributed learning tasks.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] A distributed learning task scheduling method for heterogeneous computing clusters includes:

[0008] Obtain the distributed learning task and its total computing resource request, and match each subtask in the distributed learning task with a node pool that meets the computing resource request;

[0009] Based on the node pool matched to each subtask, determine whether there is a single node that satisfies the total computing resource request of the scheduled task;

[0010] If so, then for each individual node that meets the total computing resource request of the scheduled task, a score is given using the set scheduling objective function, and the individual node with the highest score is selected to be assigned a distributed learning task; whereby the scheduling objective function includes three scheduling objectives: computing performance, communication overhead, and resource concentration.

[0011] If not, cross-node scheduling is performed: the search space size is calculated by combining the node pools matched by each subtask. When the search space size is not greater than a set threshold, a global search for allocation schemes is completed using a depth-first search scheduling strategy with a greedy approach. The best scheme is then selected as the scheduling scheme from all globally searched allocation schemes, based on the set scheduling objective function. When the search space size is greater than the set threshold, a local search for allocation schemes is completed using an improved metaheuristic algorithm. The best scheme is then selected as the scheduling scheme from all locally searched allocation schemes, based on the set scheduling objective function.

[0012] A distributed learning task scheduling system for heterogeneous computing clusters includes:

[0013] The task acquisition and node pool matching unit is used to acquire distributed learning tasks and their total computing resource requests, and to match node pools that meet the computing resource requests for each subtask in the distributed learning task.

[0014] The condition judgment unit is used to determine whether there is a single node that satisfies the total computing resource request of the scheduled task based on the node pool matched by each subtask.

[0015] A single-node scheduling unit is used to score all single nodes that meet the total computing resource requirements of the scheduling task when there is a single node that meets the total computing resource requirements of the scheduling task. The single node with the highest score is then selected to be assigned a distributed learning task. The scheduling objective function includes three scheduling objectives: computing performance, communication overhead, and resource concentration.

[0016] The cross-node scheduling unit is used to perform cross-node scheduling when no single node can satisfy the total computing resource request of the scheduled task. It calculates the solution search space size by combining the node pools matched to each subtask. When the search space size is not greater than a set threshold, a global search for allocation schemes is completed using a depth-first search scheduling strategy with a greedy approach. The optimal scheme is then selected from all globally searched allocation schemes based on a set scheduling objective function. When the search space size is greater than the set threshold, a local search for allocation schemes is completed using an improved metaheuristic algorithm. The optimal scheme is then selected from all locally searched allocation schemes based on a set scheduling objective function.

[0017] A processing device includes: one or more processors; and a memory for storing one or more programs;

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0019] A readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0020] As can be seen from the technical solution provided by the present invention, for distributed learning and training tasks, a scheduling objective function that satisfies the characteristics of distributed learning tasks is designed. This function comprehensively considers the computational performance, communication overhead, and resource utilization of the scheduling scheme to achieve a balanced scheduling among the different characteristics of distributed learning tasks. At the same time, when scheduling across nodes, an appropriate scheduling strategy is selected based on the size of the corresponding solution search space. The search overhead of the allocation scheme and the specific score of the allocation scheme are comprehensively considered during the scheduling process. Ultimately, the optimal scheduling scheme is obtained with a smaller search overhead while satisfying the characteristics of distributed learning tasks, thereby achieving the effect of accelerating distributed learning tasks. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart illustrating a distributed learning task scheduling method for heterogeneous computing clusters provided in this embodiment of the invention;

[0023] Figure 2 This is a schematic diagram of a distributed learning task scheduling system for heterogeneous computing clusters provided in an embodiment of the present invention;

[0024] Figure 3 This is a schematic diagram of a processing device provided in an embodiment of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0026] First, the following explanations are provided for the terms that may be used in this article:

[0027] The terms “including,” “comprising,” “containing,” “having,” or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, “including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.)” should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0028] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0029] The following provides a detailed description of a distributed learning task scheduling method and system for heterogeneous computing clusters provided by this invention. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, they should be performed according to conventional conditions in the art or conditions recommended by the manufacturer.

[0030] Example 1

[0031] This invention provides a distributed learning task scheduling method for heterogeneous computing clusters, which mainly includes the following steps:

[0032] Step 1: Obtain the distributed learning task and its total computing resource request, and match each subtask in the distributed learning task with a node pool that meets the computing resource request.

[0033] Step 2: Based on the node pool matched to each subtask, determine whether there is a single node that satisfies the total computing resource request of the scheduled task; if yes, proceed to step 3; if no, proceed to step 4.

[0034] Step 3: For each individual node that meets the total computing resource request of the scheduled task, score it using the set scheduling objective function, and select the individual node with the highest score to assign the distributed learning task to it; where the scheduling objective function includes three scheduling objectives: computing performance, communication overhead, and resource concentration.

[0035] Step 4: Perform cross-node scheduling: Calculate the solution search space size by combining the node pools matched to each subtask. When the search space size is not greater than a set threshold, a global search for allocation schemes is completed using a depth-first search scheduling strategy with a greedy approach. The best scheme is then selected as the scheduling scheme from all globally searched allocation schemes, based on the set scheduling objective function. When the search space size is greater than the set threshold, a local search for allocation schemes is completed using an improved metaheuristic algorithm. The best scheme is then selected as the scheduling scheme from all locally searched allocation schemes, based on the set scheduling objective function.

[0036] To more clearly demonstrate the technical solution and its effects provided by the present invention, the method provided by the embodiments of the present invention will be described in detail below with reference to specific examples.

[0037] I. Overview of the Plan.

[0038] When a distributed learning task is submitted, the task scheduler first obtains the specific computing resource request information for the task. In traditional scheduling methods, the cluster typically uses a First-Come, First-Served (FCFS) algorithm to initially filter nodes within the cluster according to the order of subtask requests within the training task, leaving nodes that meet the corresponding subtask's computing resource requests. Then, the filtered nodes are sorted in ascending order of their available computing resources, and the subtasks are assigned to the corresponding nodes. During this scheduling process, although the training task can successfully run while meeting its own resource requests, it struggles to meet the high computing performance and low communication overhead required by the distributed learning task, thus drastically increasing the completion time. To better meet the characteristics of distributed learning tasks, this paper comprehensively considers the performance of computing devices and the training communication overhead in different scheduling schemes, thereby achieving accelerated training of distributed learning tasks.

[0039] Those skilled in the art will understand that each node contains several computing devices (e.g., GPU devices).

[0040] like Figure 1 As shown, the overall process of the solution can be summarized into the following parts:

[0041] (1) Task submission.

[0042] When a distributed learning task submission request is received, the system first collects information on the training task's computational resource request, the current cluster communication overhead, and the computational resource performance. Once the minimum requirements for the training task are met, the system proceeds to specific scheduling.

[0043] (2) Task scheduling.

[0044] Taking into account multiple dimensions such as communication overhead, equipment performance, and resource utilization, the algorithm first determines whether single-node scheduling is satisfied. If it is, it enters the single-node scheduling scheme; otherwise, it enters the cross-node scheduling scheme. Based on the search space size of different allocation schemes, the algorithm selects an appropriate scheduling algorithm to balance the search overhead of the allocation scheme and the comprehensive score corresponding to the allocation scheme.

[0045] (3) Task binding.

[0046] After the scheduling scheme is selected, the specific allocation scheme is then implemented with fine-grained scheduling to ensure that the computing resource requests in the training task are allocated in sequence according to the scores of the computing devices within the node, thereby achieving optimization from coarse-grained scheduling at the node level to fine-grained scheduling at the device level.

[0047] After generating the final scheduling scheme using the above method, update the resource information in the cluster nodes and feed the scheduling scheme back to the corresponding distributed learning task (training task).

[0048] II. Detailed introduction of the plan.

[0049] 1. Scheduling objective function.

[0050] In this embodiment of the invention, a corresponding scheduling objective function is designed based on the characteristics of distributed learning tasks (training tasks). Theoretically, the scheduling objective function is expressed as:

[0051] The scheduling objective function is expressed as follows:

[0052] F(x) = γ1f perfo (x)+γ2f commu (x)+γ3f utili (x)

[0053] Where F(x) represents the scheduling objective function, x is the allocation scheme, and includes the computational performance f perfo (x), communication overhead f commu (x) and resource concentration f utili (x) Three scheduling objectives, γ1, γ2, and γ3 represent the balance factors of different scheduling objectives, which indicate the degree of emphasis of the allocation scheme on different scheduling objectives.

[0054] Specifically, regarding resource allocation details, the allocation scheme designed in this invention differs from traditional coarse-grained node-level scheduling. Instead, it adopts fine-grained device-level scheduling. The resource allocation scheme not only includes the nodes allocated to different subtasks but also precisely allocates them to specific computing devices. Therefore, different allocation schemes are represented as follows:

[0055] Gplan=gid1,gid2,...,gid ngpu

[0056] Where ngpu represents the total number of computing resource requests for a subtask, corresponding to ngpu computing devices, and gid k Let be the computing device ID corresponding to the k-th computing resource, where k = 1, 2, ..., ngpu.

[0057] The computational performance of the allocation scheme is expressed as follows:

[0058]

[0059] Among them, P(gid) k Let be the performance score of the device corresponding to the k-th computing resource. Therefore, the computing performance of this allocation scheme is determined by the sum of the performance of the allocated computing devices.

[0060] When using single-node scheduling, the communication overhead of the allocation scheme only includes the communication overhead between computing devices within the node, expressed as:

[0061]

[0062]

[0063] S l =(gid a gid b ), gid a ∈Gplan,gid b ∈Gplan

[0064] Where Comm represents the communication overhead between the two computing devices, and m represents the total number of any two computing devices selected from the ngpu computing devices. S l This represents the IDs of the two computing devices retrieved (i.e., gid). a With gid b That is, the communication overhead of the current allocation scheme is determined by the sum of the communication overhead between each pair of corresponding computing devices.

[0065] Furthermore, it should be noted that if the scheduling process involves cross-node scheduling, the communication overhead between different nodes should be added after calculating the internal communication overhead of all subtasks. node-commu(x), where the communication overhead is specifically represented as follows:

[0066]

[0067] Nplan=nid1,nid2,...,nid nodes

[0068]

[0069] N v =(nid a nid b ), nid a ∈Nplan,nid b ∈Nplan

[0070] Where Nplan represents the set of nodes assigned to all subtasks, nodes represents the number of nodes assigned after removing duplicate nodes, and nid t Let t represent the node ID corresponding to the t-th node, where t = 1, 2, ..., nodes. Let NodeComm represent the communication cost between two nodes, and n represent the total number of nodes that can be randomly selected from the nodes. N v This indicates the IDs (i.e., nids) of the two retrieved nodes. a With hid b ).

[0071] The resource concentration of the allocation scheme is expressed as:

[0072]

[0073] Here, nfree represents the amount of idle computing resources on the currently allocated nodes. In other words, the resource concentration of the current allocation scheme is determined by the ratio of idle computing resources used by nodes before and after the allocation.

[0074] The score of each subtask allocation scheme is calculated using the above scheduling objective function. Finally, the scores of all subtask allocation schemes are summed to obtain the allocation scheme score of the distributed learning task.

[0075] 2. Obtain the total computing resource request for the distributed learning task, and arrange the different subtasks within the task in order of computing resource request size (e.g., in descending order), and match each subtask in the arrangement with a node pool that meets the corresponding computing resource request.

[0076] T = T1, T2, T3, ..., T X

[0077] N = N1, N2, N3, ..., N y

[0078] NPool i =Match(T) i , N), T i ∈T

[0079] Among them, T i Let N be the sorted i-th subtask, i = 1, 2, 3, ..., X, where X is the number of subtasks and N is the number of nodes, N1, N2, N3, ..., N. y This represents the 1st, 2nd, 3rd, ..., yth node, where y is the number of nodes; NPool i For subtask T i The matching node pool, where Match is the matching function.

[0080] After completing the node pool matching, the scheduling algorithm needs to initially determine whether there is a single node with remaining computing resources that can satisfy the sum of the computing resource requests of all subtasks of the training task (i.e., the total computing resource request). If so, it enters the single-node scheduling process; otherwise, it enters the cross-node scheduling process.

[0081] 3. Single-node scheduling.

[0082] If the distributed learning task satisfies single-node scheduling, then all individual nodes that meet the total computing resource request conditions of the training task are counted. The algorithm sorts the sub-tasks in descending order according to the number of computing resource requests, and scores the sub-tasks in turn according to the scheduling objective function mentioned above. Finally, the scores of all sub-tasks are summed to obtain the total score of the entire scheduling node. Finally, the individual nodes that meet the total computing resource requirements of the training task are sorted in descending order according to the score results, and the node with the highest score is selected and assigned to the corresponding distributed learning task.

[0083] 4. Cross-node scheduling.

[0084] If single-node scheduling is not satisfied, the distributed learning task will enter cross-node scheduling. First, the size of the solution search space is calculated, expressed as:

[0085]

[0086] Among them, NPool i Subtask T i The matching node pool, where X is the number of subtasks, Len represents the length of the computation node pool, and Π is the multiplication symbol.

[0087] Determine if the following formula is satisfied:

[0088]

[0089] Where λ is the set threshold (the threshold for the solution search space).

[0090] If the above formula is satisfied, it indicates that the solution search space corresponding to the scheduling scheme is of appropriate size, and a deterministic scheduling strategy (precise search strategy) is suitable. That is, a global search for the allocation scheme is completed through a DFS (Depth-First Search) scheduling strategy with a greedy idea. Specifically: the following traversal process is performed using a greedy DFS scheduling strategy: select any node from the node pool matching the subtask and recursively perform a depth-first search until the last subtask; at the same time, during the traversal, the computing resources requested by the subtask are deducted in advance. If the allocation process encounters insufficient resources, a pruning interruption operation is performed in time to improve the search efficiency, and finally all global search allocation schemes are obtained; according to the aforementioned scheduling objective function, a comprehensive score is evaluated for each allocation scheme, and the allocation scheme with the highest score is the best scheme, which will be used as the scheduling scheme.

[0091] If the above conditions are not met, it means that the solution search space corresponding to the scheduling scheme is greater than the preset solution search space threshold, and it is suitable to adopt an approximate scheduling strategy (approximate search strategy), that is, to complete the local search of the scheduling scheme through the metaheuristic algorithm and select the best scheme among all local searches as the scheduling scheme.

[0092] In this embodiment of the invention, a metaheuristic algorithm based on simulated annealing (SA) is first employed. The algorithm first provides an initial temperature T0, an initial solution S0, and a score F(S0) of the scheduling objective function F. Each solution represents an allocation scheme, and the initial solution represents the initial allocation scheme. Randomly swapping the allocation positions of any two subtasks in the initial solution S0 represents the allocation scheme after random perturbation. If the score of the allocation scheme after random perturbation is greater than the score of the allocation scheme before random perturbation, then the allocation scheme corresponding to the current solution is updated, i.e., the update probability P = 1. If the score of the allocation scheme after random perturbation is less than the score of the allocation scheme before random perturbation, then the allocation scheme after random perturbation is selected according to the following probability P, the specific probability of which is expressed as follows:

[0093]

[0094] Among them, S old S is the solution corresponding to the allocation scheme before the random perturbation. new is the solution corresponding to the allocation scheme after random perturbation; exp is an exponential function with the natural constant as the base.

[0095] After completing one random perturbation, the initial temperature T0 is cooled down, and the next random perturbation cycle begins. The corresponding cooling strategy is as follows:

[0096] T0←η×T0,η∈(0,1)

[0097] Where η is the coefficient in the cooling strategy, and the symbol ← is the assignment symbol, indicating that the calculation result on the right is assigned to T0.

[0098] Each random perturbation is a local search process, and the updated current solution is retained; when the temperature drops to the specified temperature T... end Then, it exits the random perturbation loop and returns the current solution that has been retained after all local search processes, and uses its corresponding allocation scheme as the scheduling scheme.

[0099] Those skilled in the art will understand that temperature in the simulated annealing algorithm mainly serves to adjust the probability of accepting bad solutions; as the temperature decreases, the probability of the algorithm accepting bad solutions decreases.

[0100] Preferably, the metaheuristic algorithm of simulated annealing (SA) is improved to avoid traditional heuristic algorithms from getting trapped in local optima or bad solutions. This invention uses two optimization strategies, global optimal solution and multi-threaded concurrency, to ensure the effectiveness of the allocation scheme.

[0101] The global optimal solution S is updated after each random perturbation in the following manner. best :

[0102]

[0103] In the above formula, when the allocation scheme corresponding to the random perturbation is S new The score F(S) new (Higher than the global optimal solution S) best The score F(S) best If the global optimal solution S is obtained, then the global optimal solution S will be obtained. best Updated to S new Otherwise, like the traditional SA algorithm described above, only S is updated probabilistically. new Ultimately, by returning the globally optimal solution for scheduling, the possibility of the current solution getting stuck in a local optimum or a bad solution is reduced.

[0104] Multi-threaded concurrency is adopted, and each thread eventually returns a globally optimal solution S. best All globally optimal solutions S best The allocation scheme corresponding to the highest-scoring solution is the optimal scheme, which will be used as the scheduling scheme, denoted as:

[0105] S gbest =Sort(S pool Pop

[0106]

[0107]

[0108] in, The globally optimal solution returned by the z-th thread. z .SA indicates that the z-th thread uses the improved simulated annealing metaheuristic algorithm, S pool Sort(S) represents the set of globally optimal solutions returned by all tnum threads. pool Pop indicates that the globally optimal solutions returned by each thread are sorted in descending order of their scheduling objective function scores, and the highest-scoring solution S is popped. gbest Used as a scheduling scheme.

[0109] Therefore, by combining the above optimization strategies, we can avoid the local optima and bad solutions that are prone to occur in the traditional SA algorithm, and ensure the quality of the searched solutions.

[0110] The above-mentioned solution provided by the embodiments of the present invention mainly achieves the following beneficial effects: For distributed learning training tasks, a scheduling objective function that satisfies the characteristics of distributed learning tasks is first designed. It comprehensively considers the computational performance, communication overhead, and resource utilization of the scheduling scheme to meet the balanced scheduling among the characteristics of different distributed learning tasks. At the same time, in specific cross-node scheduling, an appropriate scheduling strategy is selected based on the size of the corresponding solution search space. The search overhead of the allocation scheme and the specific score of the allocation scheme are comprehensively considered during the scheduling process. Finally, the optimal scheduling scheme is obtained with a smaller search overhead under the premise of satisfying the characteristics of distributed learning tasks, thereby achieving the effect of accelerating distributed learning tasks.

[0111] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0112] Example 2

[0113] This invention also provides a distributed learning task scheduling system for heterogeneous computing clusters, which is mainly used to implement the methods provided in the foregoing embodiments, such as... Figure 2 As shown, the system mainly includes:

[0114] The task acquisition and node pool matching unit is used to acquire distributed learning tasks and their total computing resource requests, and to match node pools that meet the computing resource requests for each subtask in the distributed learning task.

[0115] The condition judgment unit is used to determine whether there is a single node that satisfies the total computing resource request of the scheduled task based on the node pool matched by each subtask.

[0116] A single-node scheduling unit is used to score all single nodes that meet the total computing resource requirements of the scheduling task when there is a single node that meets the total computing resource requirements of the scheduling task. The single node with the highest score is then selected to be assigned a distributed learning task. The scheduling objective function includes three scheduling objectives: computing performance, communication overhead, and resource concentration.

[0117] The cross-node scheduling unit is used to perform cross-node scheduling when no single node can satisfy the total computing resource request of the scheduled task. It calculates the solution search space size by combining the node pools matched to each subtask. When the search space size is not greater than a set threshold, a global search for allocation schemes is completed using a depth-first search scheduling strategy with a greedy approach. The optimal scheme is then selected from all globally searched allocation schemes based on a set scheduling objective function. When the search space size is greater than the set threshold, a local search for allocation schemes is completed using an improved metaheuristic algorithm. The optimal scheme is then selected from all locally searched allocation schemes based on a set scheduling objective function.

[0118] Since the specific technical details involved in the above system have been described in detail in the previous embodiments, they will not be repeated here.

[0119] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0120] Example 3

[0121] The present invention also provides a processing device, such as Figure 3 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.

[0122] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0123] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example:

[0124] Input devices can be touchscreens, image acquisition devices, physical buttons, or mice, etc.

[0125] The output device can be a display terminal;

[0126] The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.

[0127] Example 4

[0128] The present invention also provides a readable storage medium storing a computer program that, when executed by a processor, implements the method provided in the foregoing embodiments.

[0129] In this embodiment of the invention, the readable storage medium is a computer-readable storage medium and can be disposed in the aforementioned processing device, for example, as a memory in the processing device. Furthermore, the readable storage medium can also be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0130] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A distributed learning task scheduling method for heterogeneous computing clusters, characterized in that, include: Obtain the distributed learning task and its total computing resource request, and match each subtask in the distributed learning task with a node pool that meets the computing resource request; Based on the node pool matched to each subtask, determine whether there is a single node that satisfies the total computing resource request of the scheduled task; If so, then for each individual node that meets the total computing resource request of the scheduled task, a score is given using the set scheduling objective function, and the individual node with the highest score is selected to be assigned a distributed learning task; whereby the scheduling objective function includes three scheduling objectives: computing performance, communication overhead, and resource concentration. If not, cross-node scheduling is performed: the search space size is calculated by combining the node pools matched by each subtask. When the search space size is not greater than a set threshold, a global search for allocation schemes is completed using a depth-first search scheduling strategy with a greedy approach. The best scheme is then selected as the scheduling scheme from all globally searched allocation schemes, based on the set scheduling objective function. When the search space size is greater than the set threshold, a local search for allocation schemes is completed using an improved metaheuristic algorithm. The best scheme is then selected as the scheduling scheme from all locally searched allocation schemes, based on the set scheduling objective function.

2. The distributed learning task scheduling method for heterogeneous computing clusters according to claim 1, characterized in that, The step of obtaining the distributed learning task and its computing resource request, and matching each subtask in the distributed learning task with a node pool that meets the computing resource request, includes: Obtain the distributed learning task and its computing resource requests. Arrange the different subtasks within the distributed learning task in order of their computing resource requests. Match each subtask in the arrangement with a node pool that matches its computing resource request, as shown below: T=T1,T2,T3,…,T X N=N1,N2,N3,…,N y NPool i =Match(T i ,N),T i ∈T Among them, T i Let N be the i-th subtask after sorting, i = 1, 2, 3, ..., X, where X is the number of subtasks and N is the number of nodes, N1, N2, N3, ..., N. y This represents the 1st, 2nd, 3rd, ..., yth node, where y is the number of nodes; NPool i For subtask T i The matching node pool, where Match is the matching function.

3. The distributed learning task scheduling method for heterogeneous computing clusters according to claim 1, characterized in that, The size of the search space is calculated using the following formula: Among them, NPool i Subtask T i The matching node pool, where X is the number of subtasks, Len represents the length of the computation node pool, and ∏ is the multiplication symbol.

4. The distributed learning task scheduling method for heterogeneous computing clusters according to claim 1, characterized in that, The process of performing a global search for allocation schemes using a depth-first search scheduling strategy with a greedy approach, and selecting the optimal scheme from all globally searched allocation schemes based on a set scheduling objective function, includes: The following traversal process is executed using a depth-first search scheduling strategy with a greedy approach: select any node from the node pool that matches the subtask and recursively perform a depth-first search until the last subtask; at the same time, during the traversal process, resources are deducted in advance for the computation of subtask requests, and if there is insufficient resources in the allocation process, pruning is performed in time to interrupt the operation, and finally the allocation scheme of all global searches is obtained. For each allocation scheme in the global search, a set scheduling objective function is used to score it. The scheme with the highest score is the best scheme and will be used as the scheduling scheme.

5. A distributed learning task scheduling method for heterogeneous computing clusters according to claim 1, characterized in that, The step of performing a local search for allocation schemes using an improved metaheuristic algorithm, and selecting the optimal scheme from all locally searched allocation schemes as the scheduling scheme in conjunction with a set scheduling objective function, includes: The metaheuristic algorithm using simulated annealing is adopted. First, given the initial temperature T0, initial solution S0, and the score F(S0) of the scheduling objective function F, each solution represents an allocation scheme. The initial solution represents the initial allocation scheme. Randomly swapping the allocation positions of any two subtasks in the initial solution S0 represents the allocation scheme after random perturbation. If the score of the allocation scheme after random perturbation is greater than the score of the allocation scheme before random perturbation, then the allocation scheme corresponding to the current solution is updated, i.e., the update probability P = 1. If the score of the allocation scheme after random perturbation is less than the score of the allocation scheme before random perturbation, then the allocation scheme accepting the random perturbation is selected according to the following probability P, expressed as: Among them, S old S is the solution corresponding to the allocation scheme before the random perturbation. new Let be the solution corresponding to the allocation scheme after random perturbation, and exp be an exponential function with the natural constant as the base; Each random perturbation is a local search process. After completing one random perturbation, the initial temperature T0 is lowered, and the next random perturbation cycle begins. When the temperature drops to the specified temperature T, the search continues. end Then, it breaks out of the random perturbation loop, and the cooling strategy becomes: T0←η×T0,η∈(0,1) Where η is the coefficient in the cooling strategy, and the symbol ← is the assignment symbol, indicating that the calculation result on the right is assigned to T0; The metaheuristic algorithm for simulated annealing is improved by introducing two optimization strategies: global optimal solution and multi-threaded concurrency. After each random perturbation, it is determined whether to update the global optimal solution using the allocation scheme after the random perturbation. Multi-threaded concurrency is adopted to obtain the global optimal solution for all threads. The best scheme is selected from the allocation schemes of all global optimal solutions through the set scheduling objective function as the scheduling scheme.

6. A distributed learning task scheduling method for heterogeneous computing clusters according to claim 5, characterized in that, After each random perturbation, the global optimal solution S is updated as follows: best : In the above formula, when the allocation scheme corresponding to the random perturbation is S new The score F(S) new (Higher than the global optimal solution S) best The score F(S) best If the global optimal solution S is obtained, then the global optimal solution S will be obtained. best Updated to S new ; Multi-threaded concurrency is adopted, and each thread eventually returns a globally optimal solution S. best All globally optimal solutions S best The allocation scheme corresponding to the highest-scoring solution is the optimal scheme, which will be used as the scheduling scheme, denoted as: S gbest =Sort(S pool ).Pop in, The globally optimal solution returned by the z-th thread. z .SA indicates that the z-th thread uses the improved simulated annealing metaheuristic algorithm, S pool S represents the set of globally optimal solutions returned by all tnum threads. gbest Sort(S) is the solution with the highest score. pool Pop indicates that the globally optimal solutions returned by each thread are sorted in descending order according to the score of the scheduling objective function.

7. A distributed learning task scheduling method for heterogeneous computing clusters according to any one of claims 1 to 6, characterized in that, The scheduling objective function includes: computational performance f perfo (x), communication overhead f commu (x) and resource concentration f utili (x) Three scheduling objectives are represented as follows: F(x)=γ1f perfo (x)+γ2f commu (x)+γ3f utili (x) Where F(x) represents the scheduling objective function, x is the allocation scheme, and γ1, γ2, and γ3 represent the balance factors for different scheduling objectives; Computational performance f perfo (x) is represented as: Among them, gid k Let P(gid) be the computing device ID corresponding to the k-th computing resource, where k = 1, 2, ..., ngpu. k ) represents the performance score of the device corresponding to the k-th computing resource, and gpu represents the total number of computing resource requests for a subtask; When using single-node scheduling, the communication overhead f commu (x) is represented as: Where Comm represents the communication overhead between the two computing devices, m represents the total number of computing devices randomly selected from ngpu computing devices, and S l Indicates the IDs of the two computing devices retrieved; When using cross-node scheduling, add the communication overhead f between different nodes. node-commu (x), communication overhead f commu (x) is represented as: Resource concentration f utili (x) is represented as: Where nfree represents the amount of free computing resources currently allocated to the node.

8. A distributed learning task scheduling system for heterogeneous computing clusters, characterized in that, include: The task acquisition and node pool matching unit is used to acquire distributed learning tasks and their total computing resource requests, and to match node pools that meet the computing resource requests for each subtask in the distributed learning task. The condition judgment unit is used to determine whether there is a single node that satisfies the total computing resource request of the scheduled task based on the node pool matched by each subtask. A single-node scheduling unit is used to score all single nodes that meet the total computing resource requirements of the scheduling task when there is a single node that meets the total computing resource requirements of the scheduling task. The single node with the highest score is then selected to be assigned a distributed learning task. The scheduling objective function includes three scheduling objectives: computing performance, communication overhead, and resource concentration. The cross-node scheduling unit is used to perform cross-node scheduling when no single node can satisfy the total computing resource request of the scheduled task. It calculates the solution search space size by combining the node pools matched to each subtask. When the search space size is not greater than a set threshold, a global search for allocation schemes is completed using a depth-first search scheduling strategy with a greedy approach. The optimal scheme is then selected from all globally searched allocation schemes based on a set scheduling objective function. When the search space size is greater than the set threshold, a local search for allocation schemes is completed using an improved metaheuristic algorithm. The optimal scheme is then selected from all locally searched allocation schemes based on a set scheduling objective function.

9. A processing device, characterized in that, include: One or more processors; a memory for storing one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images