A method and system for scheduling online computing resources for MLaaS services

Abstract

The present invention discloses an on-line computing resource scheduling method and system for MLaaS services. The method comprises: for computing tasks entering a cluster, sequentially obtaining a computing node allocation plan based on a specified computing node allocation method, obtaining the ToR switches required for use under the allocation plan, and deriving a feasible high-level switch allocation plan for the task, wherein the computing node allocation methods include: a greedy computing node allocation method, a computing node allocation method that skips switches, a method that uses shared switches, and a computing node allocation method for shared switches; converting the global high-level switch allocation plan into a corresponding graph coloring problem, solving the problem using a simulated annealing algorithm with practical constraints, determining a reasonable global allocation plan, seamlessly switching switch resources used by running tasks according to the determined global allocation plan, and allocating switch resources to new tasks. The present invention can effectively alleviate competition for switch resources in MLaaS scenarios.

Description

Technical Field

[0001] The present invention relates to the field of network computing, and in particular to a method and system for scheduling network computing resources. Background Art

[0002] In recent years, deep neural networks have become an integral component of many applications, including computer vision, natural language processing, and systems. Distributed training across multi-rack clusters is often used to support large training datasets. Providing Machine Learning as a Service (MLaaS) is becoming a new trend. With the development of GPU hardware and the emergence of high-performance GPUs, network transmission overhead has gradually become a significant component of the training process for machine learning tasks. Numerous works have proposed solutions for accelerating machine learning training using on-network computing. These works can effectively accelerate network-sensitive machine learning tasks. Using on-network computing solutions can reduce the latency of collective operations and increase the bandwidth of collective operations.

[0003] Figure 1 This paper demonstrates the process of gradient aggregation using on-line computing for machine learning tasks. Before starting training, a switch tree needs to be established, where each node in the tree is a switch. During the training process, the computing nodes (i.e. Figure 1 Gradients calculated by the host in the diagram are split into fragments before aggregation and then aggregated on each switch along the switch tree in the diagram. Fragments with the same sequence number from different child nodes are summed at their parent switch to produce an intermediate result, which is then forwarded to higher-level switches for aggregation. The root node of the switch tree receives the final aggregated result and sends it to its child nodes. Aggregating gradients using a switch tree can reduce the traffic generated by machine learning model training by half and reduce the latency of gradient aggregation to sub-RTT levels.

[0004] Existing on-network computing methods typically share resources on the switch at a message granularity when supporting multiple tasks. However, since the switch device cache size is only about 10MB, due to differences in synchronization between task processes in 100 / 200 / 400Gbps high-speed networks, data needs to be temporarily stored on the switch during the on-network computing process. When multiple tasks share the switch's on-network computing resources, insufficient cache may occur, resulting in a slowdown in transmission. In addition, machine learning training task communications require a high degree of synchronization. A switch supporting multiple on-network computing tasks may interfere with each other. That is, two tasks share a switch, but due to competition on the switch, all nodes of both tasks need to be slowed down.

[0005] In an MLaaS environment, numerous tasks compete for switch resources. This contention can be categorized into two types: the first occurring on ToR (Top of Rack) switches, which are at the bottom of the switch tree—the leaf nodes. The second occurs on switches higher up in the tree. Both types of contention prevent some tasks from utilizing in-network computing capabilities. The first type of contention is caused by the fragmentation of compute nodes under ToR switches. Different tasks in an MLaaS data center require different numbers of compute nodes and different execution times, so compute nodes under a single ToR switch may be occupied by different jobs. This fragmentation may cause a ToR switch to handle the in-network computing traffic of multiple tasks, generating contention on the ToR switch. The second type of contention is caused by the characteristics of the switch tree used for in-network computing. Since in-network computing requires data to be stored on the switch during the computation process, the switch retains some data until the task is completed, forming a logical tree-like structure. However, due to synchronization issues involved in the job process, this data is difficult to migrate. Contention can also occur in the higher-level switch tree after jobs enter and leave due to the lack of a migration mechanism.

[0006] Furthermore, the data-partitioned distributed machine learning training process consists of two components: computation and communication. During computation, each server uses its local training data to train the model and generate new gradient updates. During communication, each server needs to obtain the gradient updates calculated by other servers and average them to obtain the overall gradient update. This step is typically performed by allreduce or a parameter server. Communication and computation are repeated throughout the training process in the form of training steps. Due to this characteristic of distributed machine learning training, network usage is intermittent and periodic, meaning that the switches occupied by the network are periodically idle during task training.

[0007] Based on the above analysis, to achieve optimal on-line computing performance, we should consider either using switch resources exclusively for tasks or sharing switch resources in new ways. This invention specifically addresses the competition between these two types of switch resources and provides an on-line computing resource scheduling solution that fully utilizes the communication characteristics of machine learning training. Summary of the Invention

[0008] Purpose of the invention: The present invention proposes an on-line computing resource scheduling method and system for MLaaS services to alleviate the competition for switch resources in MLaaS scenarios.

[0009] Technical solution: In order to achieve the above purpose, the present invention adopts the following technical solution:

[0010] In a first aspect, a method for scheduling online computing resources for MLaaS services includes the following steps:

[0011] For computing tasks that enter the cluster and need to apply for computing nodes, a computing node allocation scheme is obtained according to the greedy computing node allocation method, the computing node allocation method that skips switches, the shared switch usage method, and the shared switch computing node allocation method, and the ToR switches required under the allocation scheme are obtained, and a feasible high-level switch allocation scheme for the task is obtained, wherein the greedy computing node allocation method gives priority to allocating computing nodes under unoccupied ToR switches; the computing node allocation method that skips switches adopts a skipping method for switches with data of only one child node, transferring computing tasks to higher-level switches; the shared switch usage method allows another task to use the switch for on-line computing in the periodic gap between the computing phase of a task and the communication phase of the synchronized gradient; the shared switch computing node allocation method shares computing nodes between different tasks; the global high-level switch allocation scheme is converted into a corresponding graph coloring problem, and the problem is solved using a simulated annealing algorithm with practical constraints to determine a reasonable global allocation scheme, and the switch resources used by the running tasks are seamlessly switched according to the determined global allocation scheme, and switch resources are allocated to new tasks.

[0012] Furthermore, the computing node allocation method of skipping the switch includes:

[0013] Count the number of allocable computing nodes under each switch;

[0014] Find an unoccupied Layer 3 switch with the minimum number of allocable computing nodes and a number greater than the number of computing nodes required for the task;

[0015] The child nodes of the three-layer switch are allocated in descending order of the number of allocatable computing nodes of the child nodes and are allocated preferentially to the idle computing nodes under the occupied switch.

[0016] Furthermore, counting the number of allocable computing nodes under each switch includes:

[0017] If the current switch is not occupied, the number of allocable computing nodes is the sum of the allocable computing nodes of the lower-layer switches connected to it; if the current switch is occupied, the number of allocable computing nodes is the maximum number of allocable computing nodes of the lower-layer switches connected to it. If a ToR switch is occupied, the number of allocable computing nodes is 1; if it is not occupied, the number of allocable computing nodes is the number of idle computing nodes under the ToR switch.

[0018] Furthermore, the shared switch is used in the following ways:

[0019] When a task leaves the communication phase, it notifies the tasks shared with it that they can use the switch for on-line computing; when the task enters the communication phase, it notifies the tasks shared with it that they are prohibited from using on-line computing, and switches to using on-line computing after receiving a reply.

[0020] Furthermore, the shared switch is used in the following ways:

[0021] The control thread and the sending thread are maintained separately in multiple processes of the task. The sending thread completes the actual sending of the allreduce message. The control thread is responsible for receiving notification messages, stopping the sending thread from sending allreduce messages, and synchronizing the last allreduce message sent by other processes.

[0022] Furthermore, the computing node allocation method of the shared switch includes:

[0023] Obtain information about tasks that are currently running but not shared in the system, including the communication characteristics of each task and the number of nodes under the ToR switch occupied by the task;

[0024] Calculate the acceleration effect after sharing based on the statistics of the current task and decide which task to share with.

[0025] Prioritize allocating computing nodes under the ToR switches occupied by shared tasks, and then allocate computing nodes under unoccupied ToR switches.

[0026] Furthermore, the acceleration effects after sharing are calculated by using the statistical information of the current task, including:

[0027] Test the task before starting it, by turning on and off the on-line computation to gather information on the improvement in the communication phase of the task.

[0028] By simulating the communication phase of a new task during sharing, we found that some communication phases fell into the communication gaps of already running tasks. For the portion that fell into the gap, the communication duration when online computing was enabled was used, while for the portion that did not fall into the gap, the communication duration when online computing was not enabled was used. Through simulation, we concluded that the task received an effect of online computing acceleration when shared, and selected a model with the best acceleration effect after sharing to share with it.

[0029] Furthermore, the step of obtaining a high-level switch allocation solution that is feasible for the task includes:

[0030] After determining the assigned ToR switch, determine the connected Layer 2 switch. In a fattree network topology with k switch connections, there are k / 2 equivalent Layer 2 switch groups. For each task, select Layer 2 switches in the same position from the Layer 2 switch group to form k / 2 equivalent allocation schemes. Tasks that share switch resources are considered as one task and are assigned switches.

[0031] Furthermore, the graph coloring problem corresponding to the global high-level switch allocation solution includes:

[0032] Abstract the tasks into points. If two tasks choose the switch at the same position in the switch group, there is an edge between the two points, which is converted into a graph coloring problem with k / 2 colors.

[0033] In a second aspect, an on-line computing resource scheduling system for MLaaS services includes several switches and a global controller, wherein the several switches include ToR switches connected to computing nodes, aggregation layer switches connected to the ToR switches, and core layer switches connected to the aggregation layer switches. The global controller is configured to execute the on-line computing resource scheduling method for MLaaS services as described in the first aspect of the present invention to control the allocation of switch resources in the cluster.

[0034] The present invention has the following beneficial effects:

[0035] 1. The present invention can transfer part of the computing tasks to the high-level switches by skipping the computing node allocation method of the switch, thereby alleviating the ToR switch resource competition caused by fragmentation.

[0036] 2. The present invention can fully utilize idle online computing resources during task training by using a shared switch and allocating computing nodes, and does not affect the running tasks when sharing.

[0037] 3. The present invention generates a global switch tree allocation scheme and adjusts the resources used by tasks in the system through seamless switching without affecting task operation, thereby effectively alleviating high-level switch resource competition caused by the inability to migrate. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of using switches for network computing solutions;

[0039] Figure 2 A schematic diagram of a computing node allocation method that skips switches in the present invention;

[0040] Figure 3 Schematic diagram of a two-task shared switch in the present invention;

[0041] Figure 4 A schematic diagram of a computing node allocation method of a shared switch in the present invention;

[0042] Figure 5 A schematic diagram of a high-level switch allocation solution that is feasible for tasks in the present invention;

[0043] Figure 6 A schematic diagram of the graph coloring problem corresponding to the global high-level switch allocation solution of the present invention;

[0044] Figure 7 This is a flow chart of the on-line computing resource scheduling solution of the present invention. DETAILED DESCRIPTION

[0045] The technical solution of the present invention is further described below with reference to the accompanying drawings.

[0046] Using a switch tree to aggregate gradients can reduce the traffic generated by machine learning model training and lower the latency of gradient aggregation. Each node in the switch tree is a switch, and the switch tree has only one root switch node. The number of switch nodes used in each layer should be less than or equal to the number of switch nodes used in lower layers, and they are connected to the switch nodes in lower layers. The root switch node can be located at any layer of the physical topology, as long as the leaf nodes of the switch tree it forms are connected to all compute nodes. Switches other than the root node complete the aggregation calculation and transmit the data to the parent node. The root node completes the final data aggregation calculation and broadcasts the results along the switch tree to all compute nodes.

[0047] The process of establishing a switch tree includes: first, finding the ToR switch connected to the compute node assigned to the task. After determining the compute node, the specific ToR switch can be found; then, finding the second-layer switch (aggregation layer switch) connected to the ToR switch. There are usually multiple equivalent options, and one of the selected switches is selected in an unoccupied solution. If all ToR switches can be connected to a second-layer switch (aggregation layer switch), the root switch node has been found. Otherwise, a third-layer switch (core layer switch) connected to the second-layer switch (aggregation layer switch) is searched. At this point, a core layer switch must be found that is connected to all the second-layer switches selected according to the second-layer switch selection solution, and this core layer switch is used as the root switch node.

[0048] After obtaining the switch tree, the global controller sends messages to the switch nodes involved in the switch tree, establishes connections between switches, and allocates storage resources on the switches for tasks.

[0049] The present invention also considers the case of skipping switches. When a switch node has only one child node or compute node connected to it, it does not need to be included in the switch tree. The global controller sends messages to the switch nodes involved in the switch tree. When establishing connections between switches, the child node or compute node is logically connected directly to the higher-level switch.

[0050] The present invention also considers the situation where no feasible solution can be obtained when selecting a solution in which the selected switch is not occupied from multiple second-layer switch allocation solutions, that is, the second-layer switches selected by multiple allocation solutions are partially occupied. The present invention designs a global high-level switch allocation solution, abstracts the problem into a graph coloring problem for solution, and adjusts the switch resources used by the running tasks.

[0051] Reference Figure 2 The present invention proposes an online computing resource scheduling method for MLaaS services, comprising the following steps:

[0052] Step S1: Based on the number of computing nodes requested by the tasks entering the cluster and the number of unoccupied ToR switches, when ToR switch resources are sufficient, for example, when there are a certain number of unoccupied switches, a greedy computing node allocation method is adopted to prioritize computing nodes under ToR switches with a large number of unoccupied computing nodes, thereby allocating computing nodes to as few unoccupied ToR switches as possible and reducing ToR switch resource usage.

[0053] Step S2: When the ToR switch resources are insufficient, that is, when the greedy computing node allocation method in S1 fails, a switch-skipping computing node allocation method is adopted to transfer the computing task to a higher-layer switch.

[0054] Step S3: When the ToR switch resources are still insufficient, that is, when the computing node allocation method of skipping the switch in S2 still fails, a shared switch usage method and a shared switch computing node allocation method are adopted to improve the time dimension utilization of the switch resources;

[0055] Step S4: Obtain the required ToR switches based on the compute node allocation schemes obtained from the three compute node allocation methods of S1, S2, and S3, and derive a feasible high-level switch allocation scheme for the task.

[0056] In step S5, the global high-level switch allocation plan is generated into a corresponding graph coloring problem, and a reasonable global allocation plan is determined using a simulated annealing algorithm with practical constraints, so as to seamlessly switch the switch resources used by the running tasks and allocate switch resources for the new tasks.

[0057] When more ToR switches are occupied by more tasks, the available compute nodes under these occupied ToR switches will have difficulty utilizing on-network computing acceleration and will have to operate in a standard communication mode. Therefore, this invention implements both switch-skipping and shared-switch usage, and designs a corresponding compute node allocation scheme for these usage modes, enabling more tasks to benefit from on-network computing acceleration.

[0058] In the present invention, the global controller controls the tasks in the cluster that are already running and need to reallocate switch resources to release the switch resources they occupy. By implementing the switching logic between on-line computing and off-line computing modes, when the switch resources are reallocated to the tasks, the tasks can continue to run through the off-line computing communication method.

[0059] After all tasks have completed resource release, they report the completion of resource release to the global controller, which then notifies all tasks to reapply for resources. After resource application is complete, each task switches back to using the on-network computing communication method.

[0060] Figure 3 The figure shows the implementation of skipping switches described in the present invention, with hosts and switches in two different color depths representing different tasks. Since a switch only has data from a single child node, it doesn't actually need to calculate the gradient from that child node; instead, it can be forwarded directly to a higher-level switch. This means that the switch is skipped in the tree-like logical structure formed by the switches participating in gradient aggregation. When the global controller allocates the switch tree, it sends messages to the switch nodes involved in the tree. When establishing connections between switches, the child nodes or computation nodes of the skipped switches are logically connected directly to the higher-level switches.

[0061] When the ToR switch resources in the cluster are insufficient due to tasks, you can continue to allocate a single computing node under the occupied switches and allocate computing nodes by skipping the occupied switches, transferring computing tasks to higher-level switches. The switch-skipping computing node allocation method includes the following steps:

[0062] Step S21: Count the number of allocable computing nodes under each switch.

[0063] If the current switch is unoccupied, the number of allocable compute nodes is the sum of the allocable compute nodes of the lower-layer switches connected to it. If the current switch is occupied, the number of allocable compute nodes is the maximum of the allocable compute nodes of the lower-layer switches connected to it. If a ToR switch is occupied, the number of allocable compute nodes is 1; if it is unoccupied, the number of allocable compute nodes is the number of idle compute nodes under the ToR switch.

[0064] Step S22: Find an unoccupied three-layer switch in the physical topology (core layer switch in the fattree topology) that has the minimum number of allocable computing nodes and is greater than the number of computing nodes required to be allocated to the task.

[0065] Step S23: The child nodes (aggregation layer switches in the fattree topology) of the three-layer switch in the physical topology (core layer switches in the fattree topology) are allocated in descending order of the number of allocable computing nodes of the child nodes and the idle computing nodes under the occupied switches are allocated first.

[0066] Figure 4 The figure shows a schematic diagram of the use of the shared switch of the present invention, in which Task A and Task B conform to the communication mode of distributed machine learning model training, that is, with the iteration of training, each step of machine learning training has a calculation phase and a communication phase for synchronizing gradients, and there are periodic gaps. The use of the shared switch in the present invention is as follows: when a task leaves the communication phase, it notifies the task shared with it that it can use the switch for on-line computing; when the task enters the communication phase, it notifies the task shared with it that it prohibits the use of on-line computing, and switches to the use of on-line computing after receiving a reply. Figure 4 After Task B establishes its switch tree, a connection is established between the rank0 control threads of Task A and Task B. Task A is already running. When Task A leaves the communication phase, its rank0 process notifies the rank0 process of Task B, which it shares with it, that it can switch to using the on-net computing and transmission scheme. When Task A enters the communication phase, its rank0 process notifies the rank0 process of Task B, which it shares with it, to switch to the off-net computing scheme. After receiving the response, Task A switches to using the on-net computing and transmission scheme. Figure 4 Rank 0 / 1 / 2 are shown, representing three task processes in a distributed machine learning training task. It should be understood that in other examples or actual applications, there may be other numbers of multiple processes, each of which maintains a sending thread and a control thread.

[0067] In one embodiment, the switching between on-network computing transmission and off-network computing transmission is implemented as follows:

[0068] Two threads are maintained in each of the multiple processes of the task. The sending thread completes the actual sending of the allreduce message; the control thread is responsible for receiving notification messages, stopping the sending thread from sending allreduce messages, and synchronizing the last allreduce message sent by other processes. Allreduce is a multi-process collective communication operation. By specifying the type of calculation to be executed, such as SUM or MAX, the results stored locally by multiple processes can be summed or maximized, allowing each process to obtain the result after summing or maximizing. Allreduce is used in machine learning training tasks to sum the gradient updates of each computing node. Allreduce is an operation for collective communication among multiple processes, and the allreduce message is the specific communication content sent by each process.

[0069] When the control thread of the rank0 process receives a switch request:

[0070] Step 1: Stop the sending thread from sending the next allreduce message by locking it.

[0071] Step 2: Send a control message to other processes to stop sending allreduce messages;

[0072] Step 3: Use allgather to collect the numbers of the last allreduce messages sent by other processes. Record the largest allreduce message number + 1 as the message number of the switch. Allgather is a multi-process collective communication operation that allows multiple processes to retrieve the same variable maintained by other processes. In this case, the variable retrieved is the number of the last allreduce message.

[0073] Step 4: Release the lock to allow the sending thread of the rank 0 process to continue sending.

[0074] When the control threads of other ranks receive the switching request from rank0:

[0075] Step 1: Stop the sending thread from sending the next allreduce message by locking it.

[0076] Step 3: Use allgather to collect the numbers of the last allreduce messages sent by other processes, and record the value of the largest allreduce message number + 1 as the message number of the switch.

[0077] Step 4: Release the lock to allow the sending thread of the rank 0 process to continue sending.

[0078] Sending thread of rank0 process:

[0079] Step 1: Compare the number of the allreduce message to be sent with the number of the message to be switched obtained in the control thread. If they are equal, switching is required; otherwise, switching is not required. If switching is required, the local Flag variable is inverted.

[0080] Step 2: Notify the rank 0 node of the task to be shared.

[0081] Step 3: The thread decides whether to use the next allreduce message for on-net computing based on the Flag variable.

[0082] Sending threads of other rank processes:

[0083] Step 1: Compare the number of the allreduce message to be sent with the message number of the switching in the control thread. If they are equal, switching is required; otherwise, no switching is required. Determine whether the current message number needs to be switched. If so, invert the local Flag variable.

[0084] Step 2: The thread decides whether to use the next allreduce message for on-net computing based on the Flag variable.

[0085] Figure 5 , which shows a schematic diagram of the computing node allocation method of the shared switch described in the present invention, including the following steps:

[0086] Step 31: Obtain information about tasks that are currently running but not shared in the system, including each communication feature of the running task and the number of nodes under the ToR switch occupied by the task.

[0087] Step 32: Calculate the acceleration effect after sharing using the statistical information of the current task, and determine the task to be shared with it.

[0088] Step 33: Prioritize the allocation of computing nodes under the ToR switches occupied by the shared tasks, and then allocate computing nodes under the unoccupied ToR switches.

[0089] Furthermore, the specific method of calculating the acceleration effect after sharing by using statistical information in step 32 is:

[0090] Step a: Run a test before starting the task, briefly turning on and off the on-line computation to collect the improvement effect on the communication phase of the task.

[0091] Step b: By simulating the communication phase of a new task during sharing, we found that some communication phases fall into the communication gaps of already running tasks. For the portions that fall into the gaps, we used the communication duration when online computing was enabled. For the portions that do not fall into the gaps, we used the communication duration when online computing was not enabled. This simulation reveals the effect of online computing acceleration on the task during sharing. We select the model with the best acceleration after sharing and share it with it.

[0092] Figure 6 This example demonstrates a high-level switch usage plan generated by a task. Based on the compute node allocation plan, the required ToR switches are determined, leading to a feasible high-level switch allocation plan for the task. Once the assigned ToR switches are determined, the connected Layer 2 switches can be determined. In a fattree network topology with six switch connections, there are three equivalent Layer 2 switch groups, as shown in the box. The task can select Layer 2 switches with the same location from these three equivalent Layer 2 switch groups, forming three equivalent allocation plans. Tasks that share switch resources are treated as a single task and assigned switches.

[0093] Figure 7 It shows how to generate the corresponding graph coloring problem for the high-level switch allocation schemes of the four tasks. The global high-level switch allocation scheme generates the corresponding graph coloring problem: tasks can be abstracted into points. If two tasks select switches at the same position in the switch group, there is an edge between the two points. Equivalent two-layer switch groups are regarded as different colors, which can be converted into a graph coloring problem with k / 2 colors, where k is the number of enabled ports of the switch in the current fattree topology, that is, the number of pods in the fattree topology. Each box in the figure is an equivalent two-layer switch group, namely group1, group2, and group3. Different groups have different colors in the algorithm. In the figure, k is 6 and the number of colors is 3. If the allocation scheme shows that adjacent nodes have the same color, there will be switch resource competition in the actual allocation. Otherwise, there will be no switch resource competition. Task 4 is a task that has just entered the cluster and needs to be assigned a switch tree; Figure 7 The upper middle part shows the situation where a switch conflict occurs. Based on the above tasks, the task node diagram on the right is abstracted. Figure 7 In the example, adjacent nodes have the same color, meaning nodes 1 and 4 are both in group 2 (selecting a different group for task 4 would also result in a conflict). The lower part shows the situation after the switch conflict is resolved. Switch resources are reallocated for task 1 to use the Layer 2 switch in group 3. At this point, adjacent nodes in the task node diagram are all different colors.

[0094] For the graph coloring problem above, we can run a simulated annealing algorithm with practical constraints. The practical constraints are: there is an overhead in migrating the switch tree, and the changes in switch resource allocation for running tasks should be minimized. That is, each solution of the algorithm should be as close as possible to the previous solution.

[0095] Adjacent nodes with the same color are in conflict.

[0096] In the embodiment of the present invention, the simulated annealing algorithm process is as follows:

[0097] Step 1: All nodes are assigned colors according to the most recent assignment result. That is, the running tasks are assigned colors according to the equivalent switch set, and the new tasks are randomly assigned colors. The nodes are assigned to the color set as the initial optimal solution.

[0098] Step 2: Calculate the number of conflicts of all sets as the initial optimal conflict value.

[0099] Step 3: Group the conflicting nodes in each set into corresponding conflicting sets.

[0100] Step 4: Set the simulated annealing parameters.

[0101] Step 5: Randomly select a node from the conflicting nodes and move it to another color set to obtain a neighborhood solution as the current solution.

[0102] Step 6: Calculate the conflict value of the current solution. If it is better than the current optimal conflict value, directly accept the current solution and replace it with the optimal solution; otherwise, accept the solution with a certain probability based on the current parameters.

[0103] Step 7: Update parameters.

[0104] Step 8: Repeat steps 5-7 until the stop condition is reached.

[0105] The difference between the present invention and the existing on-line computing solutions is that the competition for switch resources is solved by scheduling on-line computing resources, and the characteristics of the on-line computing switch tree, that is, the number of switches used decreases exponentially with the number of layers, and the distance between computing nodes in the on-line computing has little effect on the allreduce bandwidth delay, are fully utilized. The fragmented computing nodes are handled by skipping the switch allocation method, and the computing tasks are transferred to the idle high-level switches for completion, effectively reducing the resource competition on the ToR switch. Unlike other methods, the message granularity sharing scheme is still limited by the switch resources and may slow down. By utilizing the communication characteristics of machine learning training, the communication gaps of machine learning tasks are fully utilized to avoid the competition caused by the simultaneous use of on-line computing resources that affects the training speed of the entire task. In addition, the present invention further reduces the competition for high-level switch resources caused by the difficulty of migration after the switch resources are allocated through a global high-level switch allocation scheme and by achieving seamless resource switching.

[0106] The present invention also provides a computer device comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the on-line computing resource scheduling method for MLaaS services as described above are implemented.

[0107] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned method for scheduling online computing resources for MLaaS services.

[0108] It will be understood by those skilled in the art that all or part of the steps to implement the above embodiments may be accomplished by hardware or by a program to instruct the relevant hardware to accomplish the steps, and the program may be stored in a computer-readable storage medium. In the context of the present invention, the computer-readable medium may be considered to be tangible and non-transitory. Non-limiting examples of non-transitory tangible computer-readable media include non-volatile memory circuits (e.g., flash memory circuits, erasable programmable read-only memory circuits, or mask read-only memory circuits), volatile memory circuits (e.g., static random access memory circuits or dynamic random access memory circuits), magnetic storage media (e.g., analog or digital tapes or hard disk drives), and optical storage media (e.g., CDs, DVDs, or Blu-ray discs), etc.

[0109] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0110] In addition, although adopting specific order to describe each operation, this should be understood as requiring such operation to be carried out in the specific order shown or in sequential order, or requiring all illustrated operations to be carried out to obtain desired result.Under certain environment, multitasking and parallel processing may be advantageous.Similarly, although comprising some specific implementation details in the above discussion, these should not be construed as limiting the scope of the present invention.Some features described in the context of independent embodiment can also be realized in single realization in combination.On the contrary, the various features described in the context of independent realization also can be realized in multiple realizations individually or in the mode of any suitable subcombination.

[0111] The preferred embodiments of the present invention are described in detail above. However, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all fall within the scope of protection of the present invention.

Claims

1. A method for scheduling online computing resources for MLaaS services, characterized in that: The following steps are involved: For computing tasks that enter a cluster and require computing nodes, a computing node allocation scheme is obtained according to the greedy computing node allocation method, the computing node allocation method that skips switches, the usage method of shared switches, and the computing node allocation method of shared switches, and the computing node allocation method of shared switches are used in turn to obtain the computing node allocation scheme, obtain the ToR switches required for the allocation scheme, and obtain a feasible high-level switch allocation scheme for the task, wherein the greedy computing node allocation method gives priority to allocating computing nodes under unoccupied ToR switches; the computing node allocation method that skips switches adopts a skipping method for switches with data of only one child node, and transfers computing tasks to higher-level switches; the computing node allocation method that skips switches includes: counting the number of allocable computing nodes under each switch, finding an unoccupied three-layer switch with the smallest number of allocable computing nodes and greater than the number of computing nodes required to be allocated for the task, and allocating the child nodes of the three-layer switch in descending order of the number of allocable computing nodes of the child nodes, with priority being given to allocating idle computing nodes under the occupied switch; the usage method of the shared switch allows another task to use the switch for on-line computing during the periodic interval between the computing phase of a task and the communication phase of the synchronous gradient; the computing node allocation method of the shared switch shares computing nodes between different tasks; The global high-level switch allocation plan is converted into a corresponding graph coloring problem. The problem is solved using a simulated annealing algorithm with practical constraints. A reasonable global allocation plan is determined. According to the determined global allocation plan, the switch resources used by the running tasks are seamlessly switched and switch resources are allocated to new tasks.

2. The method according to claim 1, characterized in that The counting of the number of allocable computing nodes under each switch includes: If the current switch is not occupied, the number of allocable computing nodes is the sum of the allocable computing nodes of the lower-layer switches connected to it; if the current switch is occupied, the number of allocable computing nodes is the maximum number of allocable computing nodes of the lower-layer switches connected to it. If a ToR switch is occupied, the number of allocable computing nodes is 1; if it is not occupied, the number of allocable computing nodes is the number of idle computing nodes under the ToR switch.

3. The method according to claim 1, characterized in that The shared switch is used in the following ways: When a task leaves the communication phase, it notifies the tasks shared with it that they can use the switch for on-line computing; when the task enters the communication phase, it notifies the tasks shared with it that they are prohibited from using on-line computing, and switches to using on-line computing after receiving a reply.

4. The method according to claim 3, characterized in that The shared switch is used in the following ways: The control thread and the sending thread are maintained separately in multiple processes of the task. The sending thread completes the actual sending of the allreduce message. The control thread is responsible for receiving notification messages, stopping the sending thread from sending allreduce messages, and synchronizing the last allreduce message sent by other processes.

5. The method according to claim 1, wherein The computing node allocation method of the shared switch includes: Obtain information about tasks that are currently running but not shared in the system, including the communication characteristics of each task and the number of nodes under the ToR switch occupied by the task; Calculate the acceleration effect after sharing based on the statistics of the current task and decide which task to share with. Prioritize allocating computing nodes under the ToR switches occupied by shared tasks, and then allocate computing nodes under unoccupied ToR switches.

6. The method according to claim 5, characterized in that The acceleration effects after sharing calculated by using the statistics of the current task include: Test the task before starting it, by turning on and off the on-line computation to gather information on the improvement in the communication phase of the task. By simulating the communication phase of a new task during sharing, we found that some communication phases fell into the communication gaps of already running tasks. For the portion that fell into the gap, the communication duration when online computing was enabled was used, while for the portion that did not fall into the gap, the communication duration when online computing was not enabled was used. Through simulation, we concluded that the task received an effect of online computing acceleration when shared, and selected a model with the best acceleration effect after sharing to share with it.

7. The method according to claim 1, characterized in that Possible high-level switch allocation solutions for this task include: After determining the assigned ToR switch, determine the connected Layer 2 switch. In a fattree network topology with k switch connections, there are k / 2 equivalent Layer 2 switch groups. For each task, select Layer 2 switches in the same position from the Layer 2 switch group to form k / 2 equivalent allocation schemes. Tasks that share switch resources are considered as one task and are assigned switches.

8. The method according to claim 1, characterized in that The graph coloring problem corresponding to the global high-level switch allocation solution includes: Abstract the tasks into points. If two tasks choose the switch at the same position in the switch group, there is an edge between the two points, which is converted into a graph coloring problem with k / 2 colors.

9. An online computing resource scheduling system for MLaaS services, characterized in that: The system comprises several switches and a global controller, wherein the several switches include ToR switches connected to computing nodes, aggregation layer switches connected to the ToR switches, and core layer switches connected to the aggregation layer switches. The global controller is configured to execute the on-line computing resource scheduling method for MLaaS services as described in any one of claims 1 to 8 to control the allocation of switch resources in the cluster.

Images