High-throughput computing resource scheduling system and method based on biological heuristic algorithm

Abstract

The invention relates to the technical field of high-throughput computing, and discloses a high-throughput computing resource scheduling system and method based on a biological heuristic algorithm, and the system comprises a task modeling and initialization module, a biological heuristic scheduling algorithm module and a dynamic adjustment and monitoring module, transmitting the processed data to a biological heuristic scheduling algorithm module; the biological heuristic scheduling algorithm module performs resource scheduling optimization and outputs a task allocation scheme to the task execution part; the dynamic adjusting and monitoring module monitors the system state in the task execution process. According to the high-throughput computing resource scheduling system and method based on the biological heuristic algorithm, a scheduling strategy based on the biological heuristic algorithm is adopted, various complex factors of tasks and resources can be comprehensively considered, the resource utilization rate is effectively increased, and the waiting time and execution time of the tasks are shortened.

Description

Technical Field

[0001] The present invention relates to the technical field of high-throughput computing, and in particular to a high-throughput computing resource scheduling system and method based on a biological heuristic algorithm. Background Art

[0002] High-throughput computing usually requires processing a large number of computing tasks and is carried out under limited computing resources. Traditional resource scheduling methods, such as First Come First Served (FCFS) and Shortest Job First (SJF), often cannot adapt well to complex and changing task requirements and resource conditions. These methods usually cannot fully consider the dependencies between tasks, the heterogeneity of resources, and the dynamic changes of the system, resulting in waste of resources and delays in task execution. Bio-inspired algorithms, such as ant colony algorithms and particle swarm optimization algorithms, have shown unique advantages in solving such complex optimization problems. However, ant colony algorithms and particle swarm optimization algorithms have not been optimized as a whole, resulting in a long time spent in actual operation, which is not convenient for the construction of a high-throughput computing framework. So far, we have proposed a high-throughput computing resource scheduling system and method based on bio-inspired algorithms. Summary of the invention

[0003] In view of the shortcomings of the existing high-throughput computing resource scheduling system and method based on biological heuristic algorithms, the present invention provides a high-throughput computing resource scheduling system and method based on biological heuristic algorithms, which has the advantages of realizing efficient and flexible resource scheduling through task modeling, algorithm optimization and dynamic adjustment, thereby achieving better resource allocation effect in high-throughput computing task processing, and solving the problems raised in the above-mentioned background technology.

[0004] The present invention provides the following technical solution: a method for high-throughput computing resource scheduling based on a biological heuristic algorithm, comprising a task modeling and initialization module, a biological heuristic scheduling algorithm module and a dynamic adjustment and monitoring module, wherein the task modeling and initialization module receives computing tasks and resource information, and transmits the processed data to the biological heuristic scheduling algorithm module; the biological heuristic scheduling algorithm module performs resource scheduling optimization, and outputs the task allocation plan to the task execution part; the dynamic adjustment and monitoring module monitors the system status during task execution, interacts with the biological heuristic scheduling algorithm module, and triggers the algorithm module to recalculate when adjustment is required.

[0005] Preferably, the task modeling and initialization module performs specific methods and parameter settings for modeling and initializing computing tasks and resources.

[0006] Preferably, the design idea of ​​improving the ant colony algorithm, the calculation method of ant transfer probability, the pheromone update mechanism and the adaptation method for high-throughput computing resource scheduling in the biological heuristic scheduling algorithm module.

[0007] Preferably, the dynamic adjustment and monitoring module includes monitoring indicators, abnormality judgment criteria, and task and resource allocation plans.

[0008] Preferably, the ant transfer probability formula is:

[0009] Assume that the transfer probability of ant k from resource node i to resource node j is The calculation formula is as follows:

[0010]

[0011] Preferably, the pheromone update formula is:

[0012] After each ant completes a traversal of the task assignment path, the pheromone on the path is updated. Assume that the pheromone volatility coefficient is ρ (0<ρ<1) and the pheromone increment is Δτ ij , then the pheromone update formula is:

[0013]

[0014] Preferably, in the ant pathfinding and resource scheduling, the ants start from the starting resource node and move between resource nodes according to the transfer probability. When the ants arrive at a certain resource node, it means that the corresponding task is assigned to the resource node for execution.

[0015] The specific operation steps of the high-throughput computing resource scheduling system based on biological heuristic algorithm are as follows;

[0016] Step 1: Create a task modeling and initialization module to collect task and resource information, establish an initial mapping, and use the biological heuristic scheduling algorithm module to run the improved ant colony algorithm;

[0017] Step 2: Observe the initial ant position and pheromone concentration, and repeat the operation until the stopping condition is met, such as reaching the maximum number of iterations or finding a satisfactory solution, and output the optimal task allocation plan;

[0018] Step 3: Start task execution, dynamically adjust and monitor the system status continuously with the monitoring module;

[0019] Step 4: Determine whether all tasks are completed. If completed, the task is completed. If not completed, return to the dynamic adjustment and monitoring module to continuously monitor the system status and continue the cycle operation;

[0020] Step 5: While the dynamic adjustment and monitoring module is continuously monitoring the system status, determine whether there are any abnormal conditions, such as resource failure, task delay, and new task addition. If it is not an abnormal state, it will return to the dynamic adjustment and monitoring module to continuously monitor the system status. If it is an abnormal state, it will adjust the task and resource allocation plan, and return to the biological heuristic scheduling algorithm module to run the improved ant colony algorithm to continue the process operation.

[0021] Compared with the existing high-throughput computing resource scheduling system and method based on biological heuristic algorithm, the present invention has the following beneficial effects:

[0022] 1. The high-throughput computing resource scheduling system and method based on the bio-inspired algorithm can comprehensively consider various complex factors of tasks and resources by adopting a scheduling strategy based on the bio-inspired algorithm, effectively improve resource utilization, and reduce task waiting time and execution time.

[0023] 2. The high-throughput computing resource scheduling system and method based on the bio-inspired algorithm enables the system to flexibly respond to various changes in the high-throughput computing process through a dynamic adjustment mechanism, thereby enhancing the stability and reliability of the system.

[0024] 3. The high-throughput computing resource scheduling system and method based on bio-inspired algorithms solves the problem of high-throughput computing resource scheduling by simulating the intelligent behavior of biological populations in nature using bio-inspired algorithms. Through task modeling, algorithm optimization and dynamic adjustment, efficient and flexible resource scheduling is achieved, thereby achieving better resource allocation effects in high-throughput computing task processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a schematic diagram of the system architecture;

[0026] Figure 2 Schematic diagram of ant path finding and resource scheduling;

[0027] Figure 3 The overall flow chart of the system. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] See also Figure 1 , Figure 2 and Figure 3A method for high-throughput computing resource scheduling based on a bio-inspired algorithm includes a task modeling and initialization module, a bio-inspired scheduling algorithm module, and a dynamic adjustment and monitoring module. The task modeling and initialization module receives computing tasks and resource information, and passes the processed data to the bio-inspired scheduling algorithm module; the bio-inspired scheduling algorithm module performs resource scheduling optimization and outputs the task allocation plan to the task execution part; the dynamic adjustment and monitoring module monitors the system status during task execution, interacts with the bio-inspired scheduling algorithm module, and triggers the algorithm module to recalculate when adjustment is needed.

[0030] The task modeling and initialization module provides specific methods and parameter settings for modeling and initializing computing tasks and resources. This module is responsible for feature extraction and modeling of high-throughput computing tasks, involving key parameters such as the amount of computing, amount of data, time limit, and dependencies of the task. At the same time, it classifies and evaluates the performance of computing resources and builds the initial mapping relationship between tasks and resources.

[0031] The design ideas of the improved ant colony algorithm, the calculation method of ant transfer probability, the pheromone update mechanism and the adaptation method for high-throughput computing resource scheduling in the biological heuristic scheduling algorithm module are used in this module to optimize resource scheduling. In this algorithm, computing resources are regarded as nodes in the ant colony, and tasks are equivalent to the food that ants need to carry. The transfer probability of ants between resource nodes will be dynamically adjusted according to factors such as resource load and task adaptability. Through the pathfinding process of ants, the optimal task and resource allocation path is continuously explored and updated. In addition, the pheromone volatilization and update mechanism is introduced to prevent the algorithm from falling into a local optimal solution.

[0032] The dynamic adjustment and monitoring module includes monitoring indicators, abnormal judgment criteria, and task and resource allocation plans. During the task execution process, the module monitors the use of computing resources, the execution progress of tasks, and the performance indicators of the system in real time. Once there are emergencies such as resource failures, task delays, or the addition of new tasks, the module will adjust the task and resource allocation plan in a timely manner and restart the bio-inspired scheduling algorithm module for optimization.

[0033] The formula for ant transfer probability is:

[0034] Assume that the transfer probability of ant k from resource node i to resource node j is The calculation formula is as follows:

[0035]

[0036] τ ij represents the pheromone concentration on the path from resource node i to resource node j, α is the pheromone importance factor, η ijrepresents the heuristic information from resource node i to resource node j, which can be defined as a comprehensive evaluation index of the adaptability of resource node j to the current task and the load of resource node j. β is the importance factor of the heuristic information, and N i Represents the set of neighbor resource nodes of resource node i.

[0037] The pheromone update formula is:

[0038] After each ant completes a traversal of the task assignment path, the pheromone on the path is updated. Assume that the pheromone volatility coefficient is ρ (0<ρ<1), and the pheromone increment is Δτ ij , then the pheromone update formula is:

[0039]

[0040] Indicates the incremental contribution of ant k to the pheromone on the path from resource node i to resource node j in this traversal. It can generally be determined according to the quality of the path found by ant k. For example, if the path found by ant k is the current optimal path, then (Q is the pheromone constant, L k is the total cost of the path taken by ant k, such as the total task execution time, etc.), otherwise

[0041] refer to Figure 2 ,Ant pathfinding and resource scheduling,The ants start from the starting resource node and move between resource nodes according to the transition probability.,When the ants reach a resource node, it means that the corresponding task will be assigned to the resource node for execution.,For example, the ant starts from resource node 1 and may move to resource node 2 or resource node 4 according to the transition probability.,If it moves to resource node 2, task 1 will be assigned to resource node 2 for execution.

[0042] refer to Figure 3 ,The system operation process of high throughput computing resource scheduling based on biological ,inspiration algorithm, the specific operation steps are as follows;

[0043] Step 1: Create a task modeling and initialization module to collect task and resource information, establish an initial mapping, and use the biological heuristic scheduling algorithm module to run the improved ant colony algorithm;

[0044] Step 3: Observe the initial ant positions and pheromone concentrations, and repeat the operation until the stopping condition is met, such as reaching the maximum number of iterations or finding a satisfactory solution, and output the optimal task allocation plan;

[0045] Step 3: Start the task execution, dynamically adjust and monitor the system status continuously

[0046] Step 4: Determine whether all tasks are completed. If completed, the task is completed. If not, return to the dynamic adjustment and monitoring module to continuously monitor the system status and continue the loop operation. During the loop operation, each ant selects a resource node according to Zhuang Yi probability, builds a task allocation path, and calculates the path cost of each ant, such as the total task execution time, and updates the global optimal path in real time. Update the pheromone concentration according to the pheromone update formula;

[0047] Step 5: While the dynamic adjustment and monitoring module is continuously monitoring the system status, determine whether there are any abnormal conditions, such as resource failure, task delay, and new task addition. If it is not an abnormal state, it will return to the dynamic adjustment and monitoring module to continuously monitor the system status. If it is an abnormal state, it will adjust the task and resource allocation plan, and return to the biological heuristic scheduling algorithm module to run the improved ant colony algorithm to continue the process operation.

[0048] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for high-throughput computing resource scheduling based on a biological heuristic algorithm, comprising a task modeling and initialization module, a biological heuristic scheduling algorithm module and a dynamic adjustment and monitoring module, characterized in that: The task modeling and initialization module receives computing tasks and resource information, and passes the processed data to the bio-inspired scheduling algorithm module; the bio-inspired scheduling algorithm module optimizes resource scheduling and outputs the task allocation plan to the task execution part; the dynamic adjustment and monitoring module monitors the system status during task execution, interacts with the bio-inspired scheduling algorithm module, and triggers the algorithm module to recalculate when adjustment is needed.

2. The method for high-throughput computing resource scheduling based on a biological heuristic algorithm according to claim 1, characterized in that: The task modeling and initialization module performs specific methods and parameter settings for modeling and initializing computing tasks and resources.

3. The method for high-throughput computing resource scheduling based on a biological heuristic algorithm according to claim 1, characterized in that: The design ideas of the improved ant colony algorithm, the calculation method of ant transfer probability, the pheromone update mechanism and the adaptation method to high-throughput computing resource scheduling in the biologically inspired scheduling algorithm module.

4. The method for high-throughput computing resource scheduling based on a biological heuristic algorithm according to claim 1, characterized in that: The dynamic adjustment and monitoring module includes monitoring indicators, abnormality judgment criteria, and task and resource allocation plans.

5. The method for high-throughput computing resource scheduling based on a biological heuristic algorithm according to claim 3, characterized in that: The formula for ant transfer probability is: Assume that the transfer probability of ant k from resource node i to resource node j is The calculation formula is as follows:

6. The method for high-throughput computing resource scheduling based on a biological heuristic algorithm according to claim 3, characterized in that: The pheromone update formula is: After each ant completes a traversal of the task assignment path, the pheromone on the path is updated. Assume that the pheromone volatility coefficient is ρ (0<ρ<1) and the pheromone increment is Δτ. ij , then the pheromone update formula is:

7. The method for high-throughput computing resource scheduling based on a biological heuristic algorithm according to claim 1, characterized in that: In the ant pathfinding and resource scheduling, it is assumed that the ants start from the starting resource node and move between resource nodes according to the transition probability. When the ants arrive at a resource node, it means that the corresponding task is assigned to the resource node for execution.

8. The system for high-throughput computing resource scheduling based on biological heuristic algorithm according to claims 1-7, characterized in that: The specific operation steps are as follows; Step 1: Create a task modeling and initialization module to collect task and resource information, establish an initial mapping, and use the biological heuristic scheduling algorithm module to run the improved ant colony algorithm; Step 2: Observe the initial ant position and pheromone concentration, and repeat the operation until the stopping condition is met, such as reaching the maximum number of iterations or finding a satisfactory solution, and output the optimal task allocation plan; Step 3: Start task execution, dynamically adjust and monitor the system status continuously with the monitoring module; Step 4: Determine whether all tasks are completed. If completed, the task is completed. If not completed, return to the dynamic adjustment and monitoring module to continuously monitor the system status and continue the cycle operation; Step 5: While the dynamic adjustment and monitoring module is continuously monitoring the system status, determine whether there are any abnormal conditions, such as resource failure, task delay, and new task addition. If it is not an abnormal state, it will return to the dynamic adjustment and monitoring module to continuously monitor the system status. If it is an abnormal state, it will adjust the task and resource allocation plan, and return to the biological heuristic scheduling algorithm module to run the improved ant colony algorithm to continue the process operation.

Images