Communication scheduling method for distributed system, and distributed machine learning system

By optimizing the communication scheduling of distributed machine learning systems in a distributed system, and using computing resource information and communication information to determine the scheduling strategy, the problem of unreasonable resource utilization between computing and communication is solved, and efficient utilization and performance of computing resources is achieved promote.

WO2024230492A1PCT designated stage expired Publication Date: 2024-11-14ALIBABA CLOUD COMPUTING CO LTD

Patent Information

Application Number
PCT/CN2024/089659
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-06
Filing Date
2024-04-24
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

In distributed machine learning systems, how to coordinate the use of computing resources between computing and communication to improve system performance, especially the problems of increasing communication time and reducing computing resource utilization efficiency during model parameter synchronization.

Method used

A distributed system communication scheduling method is proposed. By obtaining the computing resource information and communication information involved in the task, a scheduling strategy that optimizes the overall resource utilization efficiency of multiple computing resources is determined. Heuristic communication scheduling algorithm is adopted to base the number of computing resources on the task. and communication time, allocate priority, optimize communication sequence, and ensure efficient utilization of computing resources.

Benefits of technology

It realizes efficient utilization of computing resources such as GPUs in multi-task scenarios, improves the performance of distributed systems, reduces communication overhead, improves end-to-end training performance, and resolves task completion time and GPU utilization efficiency caused by network conflicts. Reduced problem.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024089659_14112024_PF_FP_ABST
    Figure CN2024089659_14112024_PF_FP_ABST
Patent Text Reader

Abstract

The present disclosure relates to a communication scheduling method for a distributed system, and a distributed machine learning system. The distributed system comprises a plurality of computing resources connected by means of a communication link network, wherein the plurality of computing resources are allocated to execute tasks, and computing resources allocated to the same task implement data synchronization by means of the communication link network. The method comprises: acquiring computing resource information and communication information involved in tasks; and on the basis of the computing resource information and the communication information involved in the tasks, determining a scheduling policy for communication related to the plurality of tasks, wherein the scheduling policy is determined by taking the optimization of the overall resource utilization efficiency of the plurality of computing resources as an objective. Therefore, the distributed system can implement better cooperation between computation and communication of the computing resources, such that the utilization of the computing resources is more rational and effective, thereby improving the performance of the whole distributed system.
Need to check novelty before this filing date? Find Prior Art

Description

Distributed system communication scheduling method and distributed machine learning system

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 6, 2023, with application number 202310499055.0 and application name “Distributed System Communication Scheduling Method and Distributed Machine Learning System”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present disclosure relates to the field of distributed computing, and in particular to a distributed system communication scheduling method and a distributed machine learning system. Background Art

[0003] In recent years, distributed systems have been widely used. One important application is the distributed machine learning (DML) system.

[0004] Distributed machine learning systems leverage computing resources (CPUs or GPUs) distributed across multiple servers to collaboratively train or infer machine learning models. Distributed machine learning systems effectively improve performance and support larger training data and models, making them a key focus of industry research. Many current deep learning training frameworks support distributed machine learning as a core feature.

[0005] Distributed machine learning systems mainly support two parallel modes.

[0006] The first is model parallelism. Model parallelism splits a machine learning model into several parts and runs each part on different GPUs. Model parallelism is primarily suitable for scenarios with large models.

[0007] The second is the data parallel mode, which divides the overall training data into multiple subsets and then feeds different data subsets into different GPUs for training.

[0008] For some scenarios where the model itself is relatively small but the training dataset is very large, the data parallel model is more suitable.

[0009] The execution process of a distributed machine learning system mainly consists of two steps.

[0010] The first step is calculation, which is responsible for calculating model parameters based on training data.

[0011] The second step is communication, which is responsible for loading training data and synchronizing model parameters between different computing nodes.

[0012] The performance of both communication and computation is important for DML systems.

[0013] In distributed systems, a very important issue is how to coordinate the use of computing resources between computing and communication.

[0014] Summary of the Invention

[0015] A technical problem to be solved by the present disclosure is to provide a distributed system communication scheduling solution, which can achieve better coordination between the calculation and communication of computing resources when used, for example, for model training in a distributed machine learning system, so that the distributed system can utilize computing resources more reasonably and effectively, thereby improving the performance of the entire distributed system.

[0016] According to a first aspect of the present disclosure, a distributed system communication scheduling method is provided, wherein the distributed system includes multiple computing resources connected via a communication link network, the multiple computing resources are allocated to perform tasks, and the computing resources allocated to the same task achieve data synchronization via the communication link network, the method comprising: obtaining computing resource information and communication information involved in the task; and determining a scheduling strategy for communications related to multiple tasks based on the computing resource information and communication information involved in the task, wherein the scheduling strategy is determined with the goal of optimizing the overall resource utilization efficiency of the multiple computing resources.

[0017] Optionally, the overall resource utilization efficiency is estimated based on the proportion of computing time of multiple computing resources.

[0018] Optionally, the overall resource utilization efficiency is estimated based on a ratio of the computing time of the plurality of computing resources to the communication time.

[0019] Optionally, the scheduling strategy is constrained by the bandwidth of each communication link and the communication bandwidth requirements of each task on each communication link.

[0020] Optionally, the step of determining a scheduling strategy for communications related to multiple tasks includes: determining the communication priority of the task based on the computing resource information and communication information involved in the task and the communication demand information on each communication link; and determining a scheduling strategy based on the communication priority, wherein, on each communication link, communications related to tasks with higher priorities are preferentially executed.

[0021] Optionally, the step of determining the communication priority of the task includes: determining the task link weight of each task relative to the communication link used by it based on the computing resource information and communication information involved in the task; and for the tasks to be assigned that have not yet been assigned a communication priority, assigning the lowest unassigned communication priority to the task with the lowest task link weight on the communication link with the largest total communication load related to the tasks to be assigned among all communication links.

[0022] Optionally, the task link weight is positively correlated with the amount of computing resources used by the task.

[0023] Optionally, the task link weight is negatively correlated with the computing resource computing time of the task.

[0024] Optionally, the task link weight is negatively correlated with the computing resource communication time of the task on the communication link.

[0025] According to a second aspect of the present disclosure, a distributed machine learning model training method is provided, including: allocating computing resources to machine learning model training tasks; dividing the training sample set of the machine learning model into multiple training sample subsets, and providing them to multiple computing resources allocated to the machine learning model training tasks respectively; multiple computing resources use the training sample subsets to train the machine learning model respectively to obtain trained model parameters; based on the computing resource information and communication information involved in the machine learning model training task, determining a scheduling strategy for communications related to multiple tasks, wherein the scheduling strategy is determined with the goal of optimizing the overall resource utilization efficiency of multiple computing resources; the computing resources allocated to the machine learning model training task communicate through a communication link network according to the scheduling strategy to achieve synchronization of model parameters.

[0026] According to a third aspect of the present disclosure, a distributed machine learning system is provided, comprising: a plurality of computing resources connected via a communication link network, the plurality of computing resources being allocated to execute training tasks of a machine learning model, the computing resources allocated to the same task communicating via the communication link network to synchronize model parameters obtained through training; and a communication scheduling system, which determines a scheduling strategy for communications related to a plurality of tasks based on computing resource information and communication information involved in the tasks, wherein the scheduling strategy is determined with the goal of optimizing the overall resource utilization efficiency of the plurality of computing resources.

[0027] According to a fourth aspect of the present disclosure, a computing device is provided, comprising: a processor; and a memory on which executable code is stored, wherein when the executable code is executed by the processor, the processor executes the method of the first aspect described above.

[0028] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising an executable code. When the executable code is executed by a processor of an electronic device, the processor is caused to execute the method as described in the first aspect above.

[0029] According to a sixth aspect of the present disclosure, a non-transitory machine-readable storage medium is provided, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor executes the method described in the first aspect above.

[0030] Therefore, the present disclosure proposes for the first time in this field to determine the communication scheduling strategy with the overall resource utilization efficiency of global computing resources such as GPU as the optimization goal, so that the distributed system can achieve better coordination between computing and communication of computing resources such as GPU, make more reasonable and effective use of computing resources, and improve the performance of the entire distributed system. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings, wherein like reference numerals generally represent like components in the exemplary embodiments of the present disclosure.

[0032] FIG1 shows a distributed machine learning system built by, for example, a cloud service provider.

[0033] FIG2 schematically shows the internal structure of a GPU server.

[0034] FIG3 schematically shows a distributed system architecture diagram of the present disclosure.

[0035] FIG4 schematically illustrates the task-related communication scheduling process of the present disclosure.

[0036] FIG5 is a schematic flowchart of a distributed machine learning model training method according to the present disclosure.

[0037] FIG6 is a schematic flowchart of a distributed system communication scheduling method according to the present disclosure.

[0038] FIG7 is a schematic flowchart of a method for determining a scheduling strategy according to an embodiment of the present disclosure.

[0039] FIG8 is a schematic flowchart of a method for determining the communication priority of a task according to an embodiment of the present disclosure.

[0040] FIG9 shows a schematic structural diagram of a computing device that can be used to implement the above method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. Rather, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0042] The technical solutions disclosed in this paper mainly focus on issues related to distributed machine learning systems in data parallel mode. It should be understood that in some scenarios, they can also be applied to distributed machine learning systems in model parallel mode.

[0043] Figure 1 shows a distributed machine learning system (DML cluster) built by, for example, a cloud service provider, which can be used to provide computing resources to customers.

[0044] In the context of this disclosure, “distributed machine learning system,” “DML system,” and “DML cluster” mean the same thing.

[0045] In addition, in the context of this disclosure, "computing resources" may include CPU and GPU.

[0046] As the focus of this disclosure, "computing resources" may only include GPUs, considering that the cost of GPUs is currently higher than that of CPUs. However, it should be understood that in some scenarios, the computing resources of this disclosure may also include CPUs.

[0047] A DML cluster can consist of multiple GPU servers and a communication link network.

[0048] Each GPU server is interconnected through multiple network switches (inter-machine switches), and there are often multiple available communication link paths between GPU servers.

[0049] FIG2 schematically shows the internal structure of a GPU server.

[0050] The CPU, GPU, and network card are connected through an internal switch, such as a PCIe (PCIe) switch, to form an internal network.

[0051] Different DML cluster solutions may have different intra-machine networks and inter-machine networks. Figures 1 and 2 only illustrate common solutions, but the solutions supported by this disclosure are not limited to intra-machine networks and inter-machine networks.

[0052] As mentioned above, in a distributed machine learning system, the computing resources allocated to each machine learning model training task, on the one hand, perform calculations, use the training sample subset to train the machine learning model, and obtain the trained model parameters; on the other hand, communicate, load the training sample subset data, and synchronize the model parameters through communication between various computing resources.

[0053] During the communication process, model parameter synchronization accounts for a very high proportion and has a significant impact on the overall system communication performance. This disclosure focuses on the model parameter synchronization part in DML communication.

[0054] Generally, model parameter synchronization needs to be performed after each round of training is completed.

[0055] There are many ways to synchronize models.

[0056] For example, for each model parameter, the maximum value of the corresponding parameter obtained after training of all computing resources can be selected, or the average value of the corresponding parameter can be taken as the model parameter to be synchronized to all computing resources.

[0057] The various computing resources can communicate directly with each other to synchronize model parameters. Alternatively, each computing resource can obtain model parameters from multiple computing resources assigned to the same training task through a unified or decentralized model parameter synchronization module, determine the model parameter values ​​to be synchronized to each computing resource, and then send the determined model parameter values ​​to each computing resource.

[0058] On the other hand, a DML cluster can simultaneously deploy multiple DML tasks. These tasks are deployed on different GPUs or CPUs, but their parameter synchronization communication processes compete for intra-machine network bandwidth and network bandwidth. This can lead to two types of conflicts: intra-machine network conflicts and inter-machine network conflicts.

[0059] The two types of conflict problems mentioned above are essentially communication network conflicts, which will increase communication time and thus increase the completion time of DML tasks.

[0060] During the communication process, computing resources such as GPUs are often in an idle waiting state. Therefore, the above two types of communication network conflict problems will also cause computing resources such as GPUs to be in an idle waiting state for a long time, thereby reducing the utilization efficiency of computing resources such as GPUs.

[0061] GPU utilization efficiency and DML task completion time, especially GPU utilization efficiency, are important performance indicators of DML clusters.

[0062] Based on the above considerations, the present disclosure proposes a distributed machine learning solution to solve the problems of increased task completion time and reduced GPU utilization efficiency caused by intra-machine and / or inter-machine network conflicts.

[0063] More specifically, this disclosure proposes a distributed machine learning (DML) communication scheduling solution. This DML communication scheduling solution can be implemented in the software layer of a distributed machine learning system server to primarily address communication conflicts in multi-tasking scenarios, such as intra-machine and inter-machine network conflicts.

[0064] The plan mainly involves two aspects.

[0065] On the one hand, we propose an optimization model that focuses on overall GPU resource utilization efficiency. This optimization model is designed for GPU resource utilization efficiency in distributed machine learning multi-task scenarios. This model can be used to measure overall GPU resource utilization efficiency when multiple DML tasks exist in a DML cluster.

[0066] On the other hand, a heuristic communication scheduling algorithm is proposed to implement this optimization model. This algorithm schedules the communication order of DML tasks in the entire DML cluster based on the GPU resource utilization efficiency model and real-time collected DML task information.

[0067] FIG3 schematically shows a distributed system architecture diagram of the present disclosure.

[0068] The distributed system architecture includes multiple GPU servers. As shown in Figure 1, multiple GPU servers can be interconnected via multiple network switches. Multiple communication link paths are often available between GPU servers.

[0069] Each GPU server may include a CPU and one or more GPUs (only one GPU is shown in each GPU server in the figure, but it should be understood that the number of GPUs is not limited to one).

[0070] Both the GPU and the CPU include model calculation modules, which can be respectively referred to as the "first model calculation module" and the "second model calculation module", which are used to perform calculations on the model training task and obtain the trained model parameters.

[0071] Both the GPU and CPU include communication proxy modules, referred to as the "first communication proxy module" and the "second communication proxy module," respectively, which communicate with the CPU or GPU on which they reside. Both the first communication proxy module integrated on the GPU and the second communication proxy module integrated on the CPU can communicate with other processes (or other GPUs or CPUs) during model training tasks.

[0072] In addition, the CPU may also include a communication scheduling module. The communication scheduling modules of multiple GPU servers may jointly generate a scheduling strategy to perform communication scheduling.

[0073] The communication agent module receives the scheduling strategy from the communication scheduling module and performs corresponding communication based on the scheduling strategy to achieve synchronization of model parameters.

[0074] When a training task assigned to computing resources initiates a new communication request, the communication agent module can obtain task-related information, such as computing resource information involved in the task (such as GPU number) and communication information (such as communication message size), and register it with the communication scheduling module.

[0075] The communication scheduling module can be a daemon process running on the CPU of each GPU server in the cluster.

[0076] The communication scheduling module can collect task information and communication link network information from the communication agent module. The above information is synchronized among all communication scheduling modules in the cluster and used to assign communication priorities.

[0077] The communication scheduling module determines a scheduling strategy for communications related to multiple training tasks currently being executed by the DML system based on the computing resource and communication information associated with each task. This scheduling strategy is determined with the goal of optimizing the overall resource utilization efficiency of these multiple computing resources. The generation of this scheduling strategy is described in detail below.

[0078] After receiving the scheduling result from the communication scheduling module, the communication agent module controls the communication by starting the communication at the start time determined based on the scheduling policy or directing the flow to a designated communication link.

[0079] FIG4 schematically illustrates the task-related communication scheduling process of the present disclosure.

[0080] First, in step S410, when a new task is started, the communication agent module of the relevant GPU or CPU collects task-related information, including the number of GPUs allocated to the task, the size of the communication message, and the GPU nodes or CPU nodes participating in the communication.

[0081] The communication agent module can register new tasks to the communication scheduling module and report the above information.

[0082] In step S420 , the communication scheduling module may calculate the priority of task communication based on information of all tasks.

[0083] In step S430, the task initiates a communication request through the communication agent module and sends the communication request to the communication scheduling module.

[0084] In step S440, the communication scheduling module outputs a scheduling policy based on the current communication status. Specifically, if a higher-priority task is already communicating and occupies the same link as the current task, the current task must wait for the higher-priority task to complete its communication before continuing. Otherwise, the current task can continue communicating directly. The generation of a plan based on the scheduling policy will be described in further detail below.

[0085] Then, in step S450, the communication agent module receives the policy from the communication scheduling module and executes the communication request according to the policy, that is, starts communication or waits.

[0086] The communication scheduling process according to the present disclosure is described above from the perspective of execution of a single task with reference to FIG. 4 .

[0087] The following describes the distributed machine learning model training method according to the present disclosure from the perspective of the entire DML system.

[0088] FIG5 is a schematic flowchart of a distributed machine learning model training method according to the present disclosure.

[0089] In step S510, computing resources are allocated to the machine learning model training tasks. For example, computing resources may be allocated to all machine learning model training tasks planned for training on the DML system at system initial startup. Alternatively, available computing resources in the DML system may be allocated to new machine learning model training tasks as they become available. One or more computing resources may be allocated to each task.

[0090] In step S520, for each machine learning model training task, the corresponding training sample set is divided into multiple training sample subsets, which are respectively provided to multiple computing resources allocated for the machine learning model training task. Each computing resource can receive one training sample subset.

[0091] In step S530, the plurality of computing resources respectively train the machine learning model (assigned thereto) using the training sample subsets to obtain trained model parameters.

[0092] In step S540, a scheduling strategy for communications related to multiple tasks can be determined based on the computing resource information and communication information involved in the machine learning model training task. The scheduling strategy is determined with the goal of optimizing the overall resource utilization efficiency of the DML system's multiple computing resources. The generation of the scheduling strategy will be described in further detail below.

[0093] In step S550, the computing resources allocated to the machine learning model training task communicate through the communication link network according to the scheduling strategy to achieve synchronization of model parameters.

[0094] Below, with reference to FIG6 , the communication scheduling method of a distributed system (e.g., a distributed machine learning system) according to the present disclosure, in particular, the planning and generation of the scheduling strategy, is further described in detail.

[0095] As previously described, the distributed system of the present disclosure includes multiple computing resources connected via a communication link network. The multiple computing resources are assigned to perform tasks, and the computing resources assigned to the same task synchronize data via the communication link network. In the case of a distributed machine learning system, the task can be a machine learning model training task.

[0096] FIG6 is a schematic flowchart of a distributed system communication scheduling method according to the present disclosure.

[0097] In step S610, computing resource information and communication information involved in the task are obtained.

[0098] For example, the communication agent module described above may collect relevant information and provide it to the communication scheduling module.

[0099] In step S620, for example, the communication scheduling module described above may determine a scheduling strategy for communications related to the multiple tasks based on the computing resource information and communication information involved in the tasks.

[0100] Different from the scheduling strategies of various existing distributed systems, the scheduling strategy disclosed herein is determined with the goal of optimizing the overall resource utilization efficiency of the multiple computing resources.

[0101] In the optimization solution of the present disclosure, the overall resource utilization efficiency may be estimated based on the proportion of computing time of multiple computing resources in the distributed system.

[0102] Alternatively, the overall resource utilization efficiency may also be estimated based on the ratio of the computing time to the communication time of the plurality of computing resources.

[0103] Considering the high cost of computing resources, especially GPUs, as mentioned above, allowing multiple computing resources in a distributed system to spend a higher proportion of their time on computing rather than waiting during communication can improve the overall resource utilization efficiency of computing resources, thereby improving the efficiency of the entire distributed system and enhancing the user experience.

[0104] Such a scheduling strategy can be represented by a computing resource (such as GPU) utilization efficiency model in a multi-tasking scenario.

[0105] Assume there are J tasks in the cluster. Each task is completed by multiple GPUs or CPUs. Each CPU or GPU runs a worker.

[0106] During each communication, the worker r uses p j,r Gbps to its peer workers (workers performing the same task) with m j,r Bytes of message.

[0107] Use c j,i and t j,i represents the computation time and communication time of task j in the i-th communication round (or i-th iteration).

[0108] In the context of machine learning model training tasks, it can generally be observed that c j,i is almost identical in each iteration and can be written with c j replace.

[0109] On the other hand, you can use t j To represent the average communication time of the total K iterations of the training task execution, that is,

[0110] This disclosure defines computing resource utilization efficiency as the average proportion of computing time to the total training time (total task execution time). As mentioned above, this is because a larger proportion of computing time indicates higher resource utilization efficiency of computing resources such as GPUs.

[0111] Therefore, the computing resource (such as GPU) utilization efficiency of each worker machine of task j can be expressed as Assume that task j is in g j The computing resource (such as GPU) utilization efficiency of task j is

[0112] In order to maximize the resource utilization efficiency of multi-task computing resources (such as GPU), the optimization goal of the computing resource (such as GPU) resource utilization efficiency model proposed in this disclosure can be expressed as the following equation:

[0113] Here, the overall resource utilization efficiency may be estimated based on the proportion of computing time of multiple computing resources in the distributed system.

[0114] During training, multiple tasks can compete for communication resources on both the intra-machine and inter-machine networks. Therefore, the scheduling strategy can be constrained by the bandwidth of each communication link and the communication bandwidth requirements of each task on each communication link.

[0115] Specifically, the above optimization objectives can be solved under the following constraints.

[0116] First, for each intra-machine link (PCIe link) l, the bandwidth p allocated to all J tasks j is j,l The sum should be less than the bandwidth capacity b of the intra-machine (PCIe) communication link l l :

[0117] Here, x represents time or time series.

[0118] Secondly, for each network link l, the task throughput p assigned to the network communication link j,l The sum should be less than the bandwidth capacity b of the network communication link l:

[0119] As above, x represents time or time series.

[0120] Finally, task communication should be able to j,i Completed within:

[0121] As above, x represents time or time series.

[0122] The objective optimization function of the resource utilization efficiency model disclosed in the present invention fully considers the limitations of the intra-machine network and the limitations of the inter-machine network, which is the first time this has been proposed in the industry.

[0123] It should be understood that in the technical solution of the present disclosure, the same strategy can be used for scheduling processing for the intra-machine (PCIe) communication link and the network communication link.

[0124] In order to reduce the complexity of the problem, the objective function can be simplified.

[0125] For example, by Taylor expanding the above optimization objective function (1), considering that the communication time is often less than the calculation time, the high-order terms in the Taylor expansion can be further removed, and the following formula can be obtained to express the optimization objective function:

[0126] In this way, the overall resource utilization efficiency can also be estimated based on the ratio of the computing time of multiple computing resources to the communication time.

[0127] By further simplifying the target optimization function with the help of Taylor expansion, a reference can be provided for the heuristic algorithm described below.

[0128] The following further describes the heuristic communication scheduling algorithm that can be used by the present disclosure to implement the aforementioned optimization model.

[0129] Heuristic algorithms are algorithms constructed based on intuition or experience, which provide a feasible solution to the combinatorial optimization problem within an acceptable cost (time and space).

[0130] In the solution of the present disclosure, multiple computing resources such as GPUs can be allocated to a task.

[0131] Each task can be assigned a strict communication priority. When communicating, each task will have exclusive access to the communication connection. In other words, a communication link can be dedicated to a single task at any given time. On each communication link, lower-priority tasks can only begin communicating after all communications with higher-priority tasks have completed.

[0132] The present disclosure proposes a priority allocation principle that prioritizes resource utilization efficiency of computing resources such as GPUs, that is, tasks that occupy more computing resources such as GPUs should be assigned higher priorities.

[0133] Due to the synchronous communication mode in data-parallel machine learning model training, it is still difficult to assign priorities to multiple tasks.

[0134] Each task has multiple workers (deployed on CPUs or GPUs) that exchange parameters at runtime (i.e., synchronize model parameters). Only when all parameters have been updated can all workers start the next round of computation and communication.

[0135] To solve this problem, the present disclosure proposes to adopt a heuristic communication scheduling algorithm, the core idea of ​​which is that the priority of a task should be equal to the lowest priority of all communication flows of the task on all paths.

[0136] Doing so ensures that all flows of the same task complete no later than the slowest communication flow of that task.

[0137] FIG7 is a schematic flowchart of a method for determining a scheduling strategy according to an embodiment of the present disclosure.

[0138] As shown in FIG. 7 , in step S710 , the communication priority of the task may be determined based on the computing resource information and communication information involved in the task, as well as the communication demand information on each communication link.

[0139] FIG8 is a schematic flowchart of a method for determining the communication priority of a task according to an embodiment of the present disclosure.

[0140] As shown in FIG8 , in step S712 , the task link weight of each task relative to the communication link used by the task may be determined based on the computing resource information and communication information involved in the task.

[0141] In an embodiment, the task link weight can be related to the amount of computing resources g used by the task j. j Positive correlation.

[0142] If a task j uses computing resources such as the number of GPUs g j Given a relatively high weight, the resource utilization efficiency of all computing resources in the entire distributed system, such as GPUs, can be improved. Therefore, the task link weight can be set to be positively correlated with the amount of computing resources used by the task.

[0143] In the embodiment, the task link weight is related to the computing resource computing time c of the task j. j Negative correlation.

[0144] If the computation time c of a task j in each round j If the waiting time is longer, then the relative impact of a certain amount of delay on the computing time of task j is small. On the contrary, if the computing time of a task j in each round is c j If is shorter, a certain amount of delay waiting time has a greater relative impact on the computing time of task j. Therefore, the task link weight can be set to be negatively correlated with the computing resource computing time of the task.

[0145] In an embodiment, the task link weight is negatively correlated with the computing resource communication time of the task on the communication link.

[0146] If the computing resource communication time of a task j is long, then further extension of the communication time will have little impact on the computing time ratio of the computing resource of the task. Therefore, the task link weight can be set to be negatively correlated with the computing resource communication time of the task.

[0147] In an embodiment, the task link weights may be set as follows.

[0148] The same task can have multiple flows of different sizes and be transmitted on different links (intra-machine communication links and / or inter-machine network communication links). Each task j should have a unique weight W on each involved communication link l. j,l .

[0149] Here, the relative communication density can be used, that is, the communication time t of task j on the communication link l j,l =m j,l / b l Divide by the communication time t in each iteration j,l and the computation time c of the computational resources of task j j The sum of the task link communication weight W of task j on link l is j,l :

[0150] The computing resources consumed by task j, such as the number of GPUs g j Divide by the computation time c j As a task-level constant weight JW j :

[0151] It can be seen that the task link weight JW j,l The amount of computing resources g used by task j j Positive correlation, the computing resource computing time c of task j j Negatively correlated with the computing resource communication time t of task j on the communication link l j,l Negative correlation.

[0152] Thus, in step S714, for the tasks to be assigned that have not yet been assigned a communication priority, the total communication load (total load to be assigned) LW related to the tasks to be assigned in all communication links can be calculated. l On the largest communication link, the task with the lowest task link weight is assigned the lowest unassigned communication priority.

[0153] Here, the total load to be distributed on link l is LWl : LW l =∑ j∈J W j,l (9)

[0154] It is represented by the task link communication ratio W of all tasks j on the link l that have not yet been assigned communication priority j,l The sum of ∑ j∈J W j,l .

[0155] In each round of communication, the link l with the highest total load (largest bottleneck) of the communication related to the task to be assigned is selected. p , and in this link l p All tasks that communicate on Find the task link weight The lowest task j = α.

[0156] Assign the lowest currently assignable priority to the task α.

[0157] Thus, as mentioned above, the priority of a task is equal to the lowest priority of all its communication flows in each communication link.

[0158] Then, the task α is excluded from the set of tasks to be assigned priorities CJ, and the task-level constant weight JW of the remaining tasks j is updated accordingly. j And the total communication load related to the tasks to be assigned on the current link l CJ←CJ\{α} (10)

[0159] Among them, "←" means assigning the value on the right to the left, that is, for CJ, JW j 、 to update.

[0160] Then, in step S716, it is determined whether there are any tasks to which no communication priority is assigned.

[0161] If there are still some, return to step S714 and continue allocating priorities.

[0162] If communication priorities have been assigned, the process returns to FIG. 7 and proceeds to step S720 .

[0163] In step S720 , a scheduling strategy may be determined based on the communication priority, wherein communications related to tasks with higher priorities are preferentially executed on each communication link.

[0164] The distributed system communication scheduling solution proposed in the present disclosure can resolve network conflicts (intra-machine network conflicts and / or inter-machine network conflicts).

[0165] By using a heuristic algorithm and taking the overall resource utilization efficiency of computing resources such as GPU as the optimization goal, the resource utilization efficiency of computing resources such as GPU for multiple tasks can be optimized as a whole, rather than just optimizing the resource utilization efficiency of a single task.

[0166] At the same time, the heuristic algorithm uses the resource utilization efficiency of computing resources such as GPU as the task weight to execute the priority allocation strategy, which can optimize the GPU resource utilization efficiency in a targeted manner rather than the communication overhead.

[0167] After experimental testing, in a multi-task scenario, the communication scheduling scheme involved in the present disclosure can bring more than 10% end-to-end training performance improvement and can reduce more than 30% of communication overhead.

[0168] The present disclosure proposes for the first time in the art a communication scheduling system for multi-tasking within a DML cluster.

[0169] This paper addresses the problem that current work cannot measure the efficiency of multi-task GPU resource utilization. Based on the characteristics of DML tasks, this paper proposes a GPU resource utilization efficiency model in multi-task scenarios for the first time based on communication time and computing time. Based on this model, it proposes a target optimization function for communication scheduling. This is the first time in this field that global GPU resource utilization efficiency is proposed as an optimization target, providing a new optimization target for DML cluster communication scheduling.

[0170] Furthermore, to address the inability of current algorithms to optimize GPU resource utilization, this paper proposes a heuristic communication scheduling algorithm based on a GPU resource utilization model in a multi-tasking scenario. This is the first communication scheduling algorithm in the field to optimize global GPU resource utilization. This algorithm efficiently assigns communication priorities to each task and effectively and collaboratively optimizes overall GPU resource utilization.

[0171] The above description uses the communication scheduling of a distributed machine learning system as an example. It should be understood that the technical solution of the present disclosure can also be applied to the communication scheduling of other distributed systems. Multiple tasks are run on a distributed system, and computing resources are allocated to each task to perform the corresponding task. The computing resources performing the same task can communicate through a communication link network to achieve data transmission or synchronization. In such a distributed system, the distributed system communication scheduling method proposed in the present disclosure can be used to determine the scheduling strategy to schedule the communication on the communication link, so as to seek a more reasonable balance between the computing and communication of computing resources and improve the overall resource utilization efficiency of computing resources.

[0172] FIG9 shows a schematic structural diagram of a computing device that can be used to implement the above method according to an embodiment of the present invention.

[0173] 9 , a computing device 900 includes a memory 910 and a processor 920 .

[0174] The processor 920 may be a multi-core processor or may include multiple processors. In some embodiments, the processor 920 may include a general-purpose main processor and one or more special coprocessors, such as a graphics processing unit (GPU) or a digital signal processor (DSP). In some embodiments, the processor 920 may be implemented using customized circuits, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs).

[0175] The memory 910 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by the processor 920 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose stored instructions and data even when the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as the permanent storage device. In other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as dynamic random access memory. The system memory may store some or all instructions and data required by the processor during operation. In addition, the memory 910 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and magnetic disks and / or optical disks may also be used. In some embodiments, the memory 910 may include a readable and / or writable removable storage device, such as a compact disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0176] The memory 910 stores executable codes. When the executable codes are processed by the processor 920 , the processor 920 can execute the method described above.

[0177] The distributed system communication scheduling method and the distributed machine learning system according to the present invention have been described in detail above with reference to the accompanying drawings.

[0178] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0179] In addition, the method according to the present invention may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing the above steps defined in the above method of the present invention.

[0180] Alternatively, the present invention can also be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or computing device, server, etc.), the processor executes the various steps of the above-mentioned method according to the present invention.

[0181] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both.

[0182] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the systems and methods according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0183] While various embodiments of the present invention have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A distributed system communication scheduling method, in, The distributed system includes a plurality of computing resources connected via a communication link network, the plurality of computing resources are allocated to perform tasks, and the computing resources allocated to the same task implement data synchronization via the communication link network, the method comprising: Obtaining computing resource information and communication information involved in the task; and Based on the computing resource information and communication information involved in the tasks, a scheduling strategy for communications related to the multiple tasks is determined, wherein the scheduling strategy is determined with the goal of optimizing the overall resource utilization efficiency of the multiple computing resources.

2. The method according to claim 1, in, The overall resource utilization efficiency is estimated based on the proportion of computing time of the multiple computing resources; or The overall resource utilization efficiency is estimated based on a ratio of the computing time to the communication time of the plurality of computing resources.

3. The method according to claim 1, in, The scheduling strategy is constrained by the bandwidth of each communication link and the communication bandwidth requirements of each task on each communication link.

4. The method according to claim 1, in, The step of determining a scheduling strategy for communications related to multiple tasks includes: Determine the communication priority of the task based on the computing resource information and communication information involved in the task and the communication demand information on each communication link; and The scheduling strategy is determined based on the communication priority, wherein, on each communication link, communications related to tasks with higher priorities are preferentially executed.

5. The method according to claim 4, in, The step of determining the communication priority of the task comprises: Determining a task link weight of each task relative to a communication link used by the task based on computing resource information and communication information involved in the task; and For the unassigned tasks that have not been assigned a communication priority, the task with the lowest task link weight on the communication link with the largest total communication load related to the unassigned tasks among all communication links is assigned the lowest unassigned communication priority.

6. The method according to claim 5, in, The task link weight is positively correlated with the amount of computing resources used by the task; and / or The task link weight is negatively correlated with the computing resource computing time of the task; and / or The task link weight is negatively correlated with the computing resource communication time of the task on the communication link.

7. The method according to claim 1, in, The distributed system is a distributed machine learning system; and / or The computing resource is a graphics processor; and / or The distributed system includes a plurality of computing servers interconnected by a network switch, wherein the computing servers include a plurality of central processing units and a plurality of graphics processing units interconnected by an in-machine switch; and / or Allocate multiple computing resources to a task; and / or Each computing resource is dedicated to a single task; and / or When communicating, a communication link is dedicated to one task at a time; and / or The task is a training task for a machine learning model. The training sample set is divided into multiple training sample subsets. Multiple computing resources allocated to the same task use a training sample subset to train the machine learning model respectively, and communicate through a communication link network to achieve synchronization of model parameters obtained through training.

8. A distributed machine learning model training method, include: Allocate computing resources for machine learning model training tasks; Divide the training sample set of the machine learning model into multiple training sample subsets, and provide them to multiple computing resources allocated for the machine learning model training task respectively; The multiple computing resources respectively use the training sample subsets to train the machine learning model to obtain trained model parameters; Determine, based on the computing resource information and communication information involved in the machine learning model training task, a scheduling strategy for communications related to the multiple tasks, wherein the scheduling strategy is determined with the goal of optimizing the overall resource utilization efficiency of the multiple computing resources; The computing resources allocated to the machine learning model training task communicate through the communication link network according to the scheduling strategy to achieve synchronization of model parameters.

9. A distributed machine learning system, include: A plurality of computing resources connected via a communication link network, wherein the plurality of computing resources are allocated to perform a training task of a machine learning model, and computing resources allocated to the same task communicate via the communication link network to synchronize model parameters obtained through training; as well as The communication scheduling system determines a scheduling strategy for communications related to multiple tasks based on computing resource information and communication information involved in the tasks, wherein the scheduling strategy is determined with the goal of optimizing the overall resource utilization efficiency of the multiple computing resources.

10. The distributed machine learning system according to claim 9, include: A plurality of graphics processing servers interconnected by a network switch, wherein the graphics processing servers include a plurality of central processing units and a plurality of graphics processors interconnected by an internal switch, and the plurality of computing resources include the plurality of graphics processing servers. Multiple graphics processors on the server.

11. The distributed machine learning system according to claim 10, in, The graphics processor includes a first model calculation module and a first communication agent module, The central processor also includes a second model calculation module, a second communication agent module and a communication scheduling module. The communication scheduling system includes a communication scheduling module of a central processing unit on the plurality of graphics processing servers, The first model calculation module and the second model calculation module are used to perform calculations of the assigned training tasks, The first communication agent module and the second communication agent module obtain computing resource information and communication information involved in the task, and register them to the communication scheduling module of the central processing unit. The communication scheduling system includes multiple communication scheduling modules that jointly generate a scheduling strategy. The communication agent module receives the scheduling strategy from the communication scheduling module, and performs corresponding communication based on the scheduling strategy to achieve synchronization of model parameters.

12. A distributed machine learning system according to any one of claims 9 to 11, in, The communication scheduling system determines the scheduling strategy by the method according to any one of claims 1 to 8.

13. A computing device, include: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 8.

14. A non-transitory machine-readable storage medium having executable codes stored thereon, which, when executed by a processor of an electronic device, causes the processor to execute the method according to any one of claims 1 to 8.

Citation Information

Patent Citations

  • Execution strategy generation method and device under cross-IDC (Internet Data Center) big data processing architecture

    CN105718479A

  • Distributed training method and device for deep learning model, and computing equipment

    CN113642734A

  • Distributed system communication scheduling method and distributed machine learning system

    CN116204327A

  • Solar panel dust detection system and method using image processing technique

    KR1020240145604A

  • Distributed computer task management of interrelated network computing tasks

    US10841236B1

Cited By

  • Distributed computing communication method and device and distributed computing method and system

    CN119182664A

  • Resource scheduling method and device for large model reasoning request

    CN119311423A

  • Embedded hybrid computing semiconductor memory

    CN119668524A

  • Task scheduling method and device, electronic equipment, storage medium and program product

    CN119718583A

  • Method for determining fragmentation strategy of model training and related device

    CN119829285A