A load balancing method and system based on machine learning assistance

Abstract

The present invention discloses a load balancing method and system based on machine learning assistance, and the method comprises: 1) an input module receives a computing task to be calculated and passes it to a prediction module; 2) the prediction module predicts the computing time required for each computing task and sends it to a planning module; 3) the planning module comprehensively plans the computing tasks on a distributed computing cluster according to the computing time: a) if a minimizing scheme is adopted, a greedy algorithm is adopted to allocate the computing tasks; b) if a min-max scheme is adopted, a greedy algorithm is adopted to allocate the computing tasks and the utilization rate P of the overall computing nodes is greater than or equal to a set threshold; 4) the computing module dynamically adjusts several computing tasks ranked later in the static planning results or computing tasks whose computing time is less than a set time length to idle computing nodes in the distributed computing cluster. The present invention can make the most of the computing power of the computing cluster.

Description

Technical Field

[0001] The present invention relates to the fields of high-performance computing, parallel computing and high-throughput computing. Specifically, in a distributed computing scenario for a large number of quantum systems or high-throughput computing tasks, a machine learning method combined with chemical informatics is used to give a predicted computing amount for each computing task. On this basis, a task planning scheme combining static and dynamic load balancing is used to balance the task load of each computing node during distributed computing, optimize the distribution of computing power and improve the efficiency of distributed computing. Background Art

[0002] The development of high-performance computing (HPC) technology has provided a solid foundation for computational simulation in the fields of materials, chemistry, and medicine. As new scientific research paradigms based on high throughput, big data, and artificial intelligence become increasingly popular, computing power planning becomes increasingly important. Reasonable scheduling and computing power allocation for a large number of computing tasks are the key to improving the overall operating efficiency of HPC computing clusters. According to Amdahl's law, it can be inferred that the theoretical parallel acceleration capability of a computer increases with the increase of computing resources. However, due to the inherent serial modules of the program and the imbalanced load during parallel operation, parallel efficiency and scalability often have bottlenecks. Compared with code-level analysis and refactoring to improve the execution efficiency of serial modules, the use of a suitable load balancing algorithm is a relatively simple solution that can greatly improve the execution efficiency of the program.

[0003] In high-throughput or first-principles calculations for materials, proteins and other systems, many computing tasks can be accelerated by parallelization or distributed computing solutions, but these computing tasks are often not necessarily the same. Therefore, a reasonable load balancing solution is very necessary to optimize the allocation of computing power and ensure the efficiency of computing. The so-called reasonable load balancing refers to the optimization strategy of allocating various tasks with different computing power requirements and different time consumption to computing resources in a reasonable manner, so as to minimize the idle computing resources in parallel or distributed computing. While optimizing resource allocation, an excellent load balancing strategy can minimize the time required for overall task calculation in a distributed computing scenario, which plays a very important role in many scenarios that require timeliness and timely computing, such as drug screening calculations, antibody molecule action calculations, etc.

[0004] However, load balancing is considered to be an NP-hard problem. Generally speaking, there are two popular load balancing algorithms, namely static load balancing and dynamic load balancing. The former does not consider the state of the computing unit during the execution of the operation, and usually requires a priori load-related data to better distinguish the task load of each computing unit; the latter needs to obtain the state of the computing unit in real time during the operation and make targeted adjustments. Generally speaking, static load balancing is simple to implement and has almost no idle time, so it is suitable for scenarios with a large number of "fine-grained" smaller computing tasks, such as the solution of Gaussian Integration in HPC scenarios. In contrast, if there are a large number of computing tasks of different sizes, dynamic load balancing is generally a more suitable solution. For example, the "work stealing" algorithm proposed by Nikodem et al. and the dynamic task allocation solution combined with tensor contraction proposed by Lai et al. In addition, there are also some dynamic load balancing frameworks such as Charm++ and cyclops tensor framework, which can also quickly implement dynamic load balancing solutions in actual calculations.

[0005] It is worth noting that when there are a large number of computing units, the execution efficiency of dynamic load balancing will be greatly affected due to the existence of information exchange in dynamic load balancing. A possible improvement is to combine dynamic load balancing with static load balancing, that is, to distribute the main tasks on the computing devices through static load technology, and then process the remaining computing tasks through dynamic load balancing technology. This improved load balancing solution has been applied to many computing fields, such as Chow et al.'s scalable Fock matrix construction solution, which applies static load and "workstealing" dynamic load solutions. In addition, there are heuristic static load balancing solutions, such as Alexeev et al.'s improved static load balancing fragmented molecular orbital calculation solution. Summary of the invention

[0006] In order to further improve the parallel computing capability, optimize task scheduling and load balancing during calculation, the present invention proposes a static and dynamic load balancing method and system that can be assisted by machine learning, and has been applied in first principle high-throughput computing. For a large number of subtasks to be calculated, each task can use 1 to n computing resources (such as computing nodes). For a large number of subtasks, machine learning can be used to batch predict the computing time required for these subtasks before calculation. After the prediction, a planning algorithm such as a greedy algorithm can be used to distribute a large number of subtasks to M computing nodes according to the predicted computing time, that is, to achieve static load planning. If the expected load of each computing node is not close to the balanced load after the initial planning, the computing resources of the task with the longest expected time are increased to significantly reduce the expected computing time of the task. On the basis of parallelizing the most time-consuming tasks at the bottom layer, the upper layer uses planning schemes such as greedy algorithms to rebalance the loads of each computing node. Obtain machine learning assisted static load planning. After the static load planning is completed, a dynamic load balancing scheme can be introduced at the same time for the tasks at the back of the task list to maximize the use of the computing power of the computing cluster.

[0007] This solution can be divided into four main modules: input module, prediction module, planning module, and calculation module. The overall process architecture is shown in the attached figure. Figure 1 As shown. We briefly describe each module as follows:

[0008] (1) Input module

[0009] This module is responsible for receiving batch calculation tasks to be calculated. This module can also perform rationality analysis, format conversion or input file generation for each calculation task. Finally, the input module passes this information to the prediction module. The calculation task is the input file required for the calculation; for quantum chemical calculations, the calculation task may include the structure of the molecule and the calculation method. The rationality analysis of quantum chemical calculations specifically includes the following in the calculation tasks: 1. The rationality of atomic bonding. If it is unreasonable, hydrogen replenishment operation can be performed. 2. Check the charge and spin multiplicity of the molecular structure in the task to be calculated. If it is unreasonable, correction can be performed. 3. Repeated check of the atomic coordinate position in the calculation task. If there is a repetition, the calculation task will be cleared from the task list and an output prompt will be given. After the above processing, the result is that a reasonable input file for the calculation task can be obtained, avoiding many errors caused by the above factors; at the same time, a reasonable input file also ensures that the prediction module can give a reliable calculation time prediction result.

[0010] (2) Prediction Module

[0011] This module is mainly responsible for predicting the computational time required for each computational task. For this part, we can use our independently developed computational time prediction module (ACSOmega_6_2001_2021) based on chemical informatics and various machine learning models.

[0012] (3) Planning module

[0013] This module mainly plans all computing tasks on the distributed computing cluster based on the computing time data provided by the prediction module. The current bottom layer mainly applies the minimizing scheme and the min-max scheme for the total computing time, which correspond to single-task single computing unit and as few parallel computing tasks as possible; the upper-level planning module adopts a greedy algorithm to continuously plan the most time-consuming tasks to the computing node with the least computing tasks, so that the expected load on each computing unit (computing node) is as close as possible. For example, for high-throughput first-principle calculations or protein subsystem calculations, the planning process is as follows: 1. Read in the input files to be calculated, such as structural data sets, fragment molecular sets, etc. 2. Store the read-in subsystem, sharding and other input files in the computer in the form of lists or tuples 3. Generate a computing task list through a program from the subsystem or shard list 4. Perform task planning according to the bottom-level planning scheme of the distributed computing cluster 5a. If the bottom layer adopts the minimizing scheme (single task single node), the outer layer uses the greedy algorithm for planning according to the predicted computing time of each computing task to obtain a static computing load planning assisted by machine learning. 5b. If the bottom layer adopts the min-max scheme (single task and multiple nodes), the outer layer first uses the greedy algorithm to plan based on the predicted computing time of each task, and calculates the expected overall node utilization P under the current plan. Where T i is the estimated computing time of the ith node, N is the total number of computing nodes, T max is the estimated computing time of the most time-consuming node. If the overall node utilization P is less than the set threshold, the number of computing nodes for the most time-consuming computing task is increased, the predicted computing time of the most time-consuming computing task is updated, and then the outer layer re-uses the greedy algorithm for planning; the cycle is executed until the overall node utilization P is greater than or equal to the set threshold, and the static computing load planning assisted by machine learning is output. See the attached Figure 2 .

[0014] (4) Computing module

[0015] The calculation module receives the static planning results from the planning module and performs calculations accordingly. During the calculation process, the calculation module can dynamically adjust several computing tasks that are ranked lower in the static planning or tasks whose computing time is less than the set time length to idle computing nodes, thus realizing task scheduling that combines static and dynamic load balancing. In order to efficiently realize real-time task scheduling that combines static and dynamic, we independently developed a distributed computing engine based on the Julia high-performance computing language and demonstrated it in Figure 3 . First, the computing engine can obtain the task scheduling authority of the cluster from popular HPC cluster task scheduling systems such as SLURM (Simple Linux Utility for Resource Management) and LSF (Load Sharing Facility); then, during the task calculation process, the engine can enable each distributed computing node to "self-monitor" the task status, that is, 1. Non-running nodes automatically obtain computing jobs according to the plan 2. Running nodes self-lock to prevent task conflicts 3. Idle nodes spontaneously match single tasks across nodes for parallel computing 4. Support heterogeneous computing. When all node tasks are completed, the computing engine is terminated and all occupied nodes are released. Compared with the default queuing scheduling systems such as SLURM and LSF, this computing engine can be regarded as a dedicated computing scheduling system with static and dynamic job scheduling capabilities. In the dynamic adjustment module of the scheduling system, two dynamic load balancing modes, top-down and bottom-up, are currently built-in, in order to better achieve the purpose of load optimization during computing, and are shown in the attached. Figure 4 . The top-down approach is to evenly distribute the tasks included in the dynamic load to each computable distributed node. At this time, the same task may be assigned to multiple computing nodes; when the computing tasks of the static planning part of the node are calculated, the uncalculated tasks will be selected from the dynamic task queue pre-assigned to this node for calculation. The bottom-up approach is to evenly distribute the tasks included in the dynamic load balancing to the task pool. When the computing tasks of the static planning part of the node are calculated, the uncalculated tasks will continue to be selected from the task pool for calculation.

[0016] The advantages of the present invention are as follows:

[0017] 1. In the scenario of distributed computing, it can significantly improve the utilization rate of nodes and reduce the vacancy rate of nodes, thereby accelerating the actual application computing.

[0018] 2. In the scenario of distributed computing, it can optimize the distribution of computing power and make real-time adjustments to improve computing efficiency and achieve the goal of energy conservation and emission reduction.

[0019] 3. The distributed computing planning scheme based on machine learning prediction and combined with dynamic and static load balancing can be promoted and applied to multiple scientific research and industrial fields and has good application potential. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is the overall architecture of the system of the present invention.

[0021] Figure 2 This is an example diagram of the static load balancing planning module process combined with machine learning calculation time prediction in the system of the present invention.

[0022] Figure 3 The figure is a schematic diagram of the implementation of the computing module applicable to static and dynamic load balancing in the system of the present invention.

[0023] Figure 4 Schematic diagram of two different dynamic load balancing modes in the system of the present invention.

[0024] Figure 5 It is a schematic diagram of the application effect of the system of the present invention;

[0025] (a) Default load planning, (b) Default load planning combined with dynamic load planning, (c) Static load planning assisted by machine learning, and (d) Static and dynamic load planning assisted by machine learning. DETAILED DESCRIPTION

[0026] The present invention is further described in detail below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention but not to limit the scope of the present invention.

[0027] The machine learning-assisted load balancing method and its implementation system described in the present invention can be implemented by Python and Julia computer programming languages. In the current implementation, we use the Julia language to implement the input module; the prediction module uses the Python-based computer prediction module (ACSOmega_6_2001_2021) developed by the research group in the early stage; the planning module is based on the data of the prediction module and is also written in Python; the calculation module is written in Julia, which mainly uses the distributed computing function of Julia, and is compatible with the cluster's task scheduling system (such as the tested SLURM, LSF, etc.). An example of the application effect of the system (the example is the calculation of the P38 protein-ligand system) is listed in the attached Figure 5 .

[0028] Although the specific embodiments of the present invention are disclosed for the purpose of illustration, the purpose is to help understand the content of the present invention and implement it accordingly, those skilled in the art will understand that various substitutions, changes and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the best embodiment, and the scope of the present invention is subject to the scope defined in the claims.

Claims

1. A load balancing method based on machine learning assistance, the steps of which include: 1) The input module receives N computational tasks to be calculated and passes them to the prediction module; N is a natural number greater than 1; 2) The prediction module predicts the computing time required for each computing task and sends it to the planning module; 3) The planning module performs overall planning of the N computing tasks on the distributed computing cluster according to the computing time provided by the prediction module: a) If the distributed computing cluster adopts a minimizing scheme of a single task and a single node, then according to the computing time provided by the prediction module, a greedy algorithm is used to allocate the N computing tasks to the corresponding computing nodes in the distributed computing cluster, and a static planning result is obtained and sent to the computing module; b) If the distributed computing cluster adopts a min-max scheme of a single task and multiple nodes, then according to the computing time provided by the prediction module, a greedy algorithm is used to allocate the N computing tasks to the corresponding computing nodes in the distributed computing cluster, and a candidate static planning is obtained, and the utilization rate P of the computing nodes under the current candidate static planning is calculated. If the utilization rate P of the overall computing nodes is less than the set threshold, the number of computing nodes of the current most time-consuming computing task is increased and the predicted computing time of the current most time-consuming computing task is updated, and then the greedy algorithm is used to reallocate the N computing tasks until the utilization rate P of the overall computing nodes in the obtained candidate static planning is greater than or equal to the set threshold, and the candidate static planning at this time is sent to the computing module as the static planning result; 4) The computing module dynamically adjusts several computing tasks that are ranked lower in the static planning results or computing tasks whose computing time is less than the set time length to idle computing nodes in the distributed computing cluster, and allocates the remaining computing tasks in the static planning results to corresponding computing nodes according to the corresponding allocation scheme in the static planning results.

2. The method according to claim 1, characterized in that The method for the calculation module to adjust the static planning result is as follows: the calculation module adds a number of calculation tasks ranked later in the static planning result or a number of calculation tasks whose calculation time is less than a set time length into a dynamic load task list; The computing tasks in the dynamic load task list are then evenly distributed to each computable computing node; when the computing node i that is assigned to the computing task according to the static planning result is also assigned the computing task in the dynamic load task list, the computing node i first completes the computing task assigned according to the static planning result, and then the computing node i completes the computing task in the assigned dynamic load task queue list.

3. The method according to claim 1, characterized in that The method for the calculation module to adjust the static planning result is as follows: the calculation module adds a number of calculation tasks ranked later in the static planning result or a number of calculation tasks whose calculation time is less than a set time length into a dynamic load task list; The computing tasks in the dynamic load task list are then evenly distributed to the task pool. When the computing node i assigned to the computing task according to the static planning result completes the assigned computing task, the uncalculated computing tasks are continuously selected from the task pool for calculation.

4. The method according to claim 1, 2 or 3, characterized in that: The computing module first obtains the task scheduling authority of the distributed computing cluster to control each computing node, so that each computing node can achieve a "self-monitoring" task state, that is, non-running computing nodes automatically obtain computing jobs according to the plan, running computing nodes self-lock to prevent task conflicts, idle nodes spontaneously match single-task cross-node parallel computing, and support heterogeneous computing.

5. The method according to claim 1, 2 or 3, characterized in that: The input module preprocesses each of the received computing tasks, and then sends the preprocessed computing tasks to the prediction module.

6. The method according to claim 5, characterized in that The calculation task is a quantum chemical calculation task, and the preprocessing includes: a) detecting the rationality of atomic bonding and supplementing hydrogen for unreasonable atomic bonding; b) checking the charge and spin multiplicity of the molecular structure in the task to be calculated, and correcting it if it is unreasonable; c) checking the atomic coordinate position.

7. A load balancing system based on machine learning assistance, characterized in that: It includes input module, prediction module, planning module and calculation module; among them, The input module is used to receive N computing tasks to be calculated and pass them to the prediction module; N is a natural number greater than 1; The prediction module is used to predict the computing time required for each computing task and send it to the planning module; the planning module is used to coordinate the planning of the N computing tasks on the distributed computing cluster according to the computing time provided by the prediction module: a) if the distributed computing cluster adopts a single-task single-node minimizing scheme, then according to the computing time provided by the prediction module, the N computing tasks are allocated to the corresponding computing nodes in the distributed computing cluster using a greedy algorithm, and a static planning result is obtained and sent to the computing module; b) if the distributed computing cluster adopts a single-task multi-node min-max scheme, then based on According to the computing time provided by the prediction module, the N computing tasks are allocated to the corresponding computing nodes in the distributed computing cluster using a greedy algorithm to obtain a candidate static plan, and the utilization rate P of each computing node under the current candidate static plan is calculated. If the utilization rate P of the overall computing node is less than a set threshold, the number of computing nodes of the current most time-consuming computing task is increased and the predicted computing time of the current most time-consuming computing task is updated. Then, the N computing tasks are reallocated using a greedy algorithm until the utilization rate P of all computing nodes in the obtained candidate static plan is greater than or equal to the set threshold, and the candidate static plan at this time is sent to the computing module as a static planning result; The computing module is used to dynamically adjust several computing tasks that are ranked lower in the static planning results or computing tasks whose computing time is less than a set time length to idle computing nodes in the distributed computing cluster, and allocate the remaining computing tasks in the static planning results to corresponding computing nodes according to the corresponding allocation scheme in the static planning results.

Images