Intelligent computing power scheduling method in distributed computing environment

Abstract

The invention provides an intelligent computing power scheduling method in a distributed computing environment, and relates to the technical field of distributed computing, and the intelligent scheduling method specifically comprises the steps of collecting resource information data, constructing a resource and task model, evaluating a computing power demand, formulating an adjustment strategy, distributing and scheduling the computing power, monitoring and adjusting in real time, and feeding back and optimizing. Through comprehensive modeling and dynamic computing power evaluation of computing nodes and tasks and in combination with multiple intelligent scheduling strategies, accurate allocation of computing power resources can be realized, the problems of resource waste and node load imbalance are effectively avoided, the overall utilization rate of resources in a distributed computing environment is remarkably improved, and the computing power of the distributed computing environment is improved according to the characteristics and requirements of the tasks. The execution nodes are reasonably selected, the resource allocation is optimized, the task correlation is considered, the data transmission overhead is reduced, the task execution speed can be increased, the task completion time can be shortened, and the processing capacity and response speed of the system are improved.

Description

Technical Field

[0001] The present invention relates to the field of distributed computing technology, and in particular to a method for intelligently scheduling computing power in a distributed computing environment. Background Art

[0002] With the rapid development of technologies such as big data, artificial intelligence, and cloud computing, distributed computing environments have been widely used in various fields. In a distributed computing environment, it is usually composed of a large number of heterogeneous computing nodes. These nodes differ in computing power, storage capacity, network bandwidth, etc., and the types and requirements of tasks are also different.

[0003] Traditional computing power scheduling methods often adopt fixed strategies, such as polling scheduling and static priority scheduling. These methods are difficult to adapt to the dynamic changes of resources and the diversity of tasks in distributed computing environments. They can easily lead to unreasonable resource allocation, resulting in some nodes being overloaded while others are idle, thereby reducing the computing efficiency and resource utilization of the entire system. To this end, we propose an intelligent computing power scheduling method in a distributed computing environment to address the above problems. Summary of the Invention

[0004] The present invention solves the problems of low computing power scheduling efficiency and low resource utilization in the prior art through dynamic evaluation and intelligent scheduling of resources and tasks, and realizes the optimal configuration of computing power resources in a distributed computing environment.

[0005] The present invention provides a method for intelligently scheduling computing power in a distributed computing environment, which comprises the following steps:

[0006] S1: Collect resource information data, which is used to collect hardware information, operating status information of computing nodes, and attribute and requirement information of tasks;

[0007] S2: Build resource models and task models for computing nodes, and transform resource and task information into mathematical models that can be calculated and analyzed;

[0008] S3: Evaluate the available computing power of computing nodes and the computing power requirements of tasks to provide a basis for scheduling decisions;

[0009] S4: Develop various scheduling strategies such as priority strategy, load balancing strategy, and task correlation strategy based on the characteristics of tasks and resources;

[0010] S5: Computing power allocation and scheduling, using optimization algorithms to perform computing power allocation and task scheduling operations, and reasonably assigning tasks to computing nodes;

[0011] S6: Monitor task execution and node operation status in real time, and make dynamic adjustments based on the monitoring results to ensure stable system operation;

[0012] S7: Feedback and optimization: collect task execution result data, optimize and improve the system model and scheduling strategy, and improve system performance;

[0013] Preferably, the resource information data specifically includes basic hardware parameters such as CPU model, number of cores, memory capacity, storage capacity, network bandwidth, etc., and the current operating status information of the node includes real-time load data such as CPU usage, memory occupancy, task queue length, disk I / O rate, network transmission rate, etc.;

[0014] The task attributes and requirement information specifically include task type, task priority, required computing resource parameters, data size, and estimated running time.

[0015] Preferably, the resource model and task model of the computing node are constructed as follows:

[0016] S201: Resource model construction: Based on the collected node information, a resource model is established to describe the resources of each computing node as a multi-dimensional vector to facilitate subsequent calculations and comparisons;

[0017] S202: Task model construction: Based on the computing requirements of the task, the task is also described as a multi-dimensional vector, corresponding to the resource model.

[0018] Preferably, the specific steps of computing power evaluation include:

[0019] S301: Node computing power evaluation: Based on the resource model, the available computing power of each node is evaluated by comprehensively considering the node's hardware performance and current load.

[0020] S302: Task computing power requirement assessment: Determine the computing power required for the task based on the task model. You can estimate the running time and resource consumption of the task under different resource configurations based on factors such as the task type and scale, combined with historical data or empirical formulas, to determine the required computing power.

[0021] Preferably, the formulation of the scheduling strategy specifically includes:

[0022] S401: Based on task priority: Set priorities for different tasks based on factors such as the importance and urgency of the tasks, and prioritize computing resources for high-priority tasks.

[0023] S402: Based on resource utilization: tasks are assigned to nodes with lower resource utilization to balance the load of the entire distributed computing environment and improve overall resource utilization;

[0024] S403: Based on task relevance: If there is relevance between tasks, such as data dependency, try to assign related tasks to the same node or nodes that are close to each other in the network topology to reduce data transmission overhead.

[0025] Preferably, the computing power allocation and scheduling specifically includes:

[0026] S501: Select nodes according to the scheduling policy: According to the established scheduling policy, the most suitable node is selected from the available computing nodes to execute the task. Various optimization algorithms, such as greedy algorithms and genetic algorithms, can be used to find the optimal allocation solution.

[0027] S502: Allocate computing resources: Allocate tasks to selected nodes and allocate corresponding computing resources to them. Rationally divide and allocate resources on the nodes to ensure that the tasks can run efficiently on the nodes.

[0028] Preferably, the real-time monitoring and adjustment specifically include:

[0029] S601: Monitoring task execution: During task execution, real-time monitoring of the task's running status, including task progress and resource usage, is performed. By monitoring the status of computing nodes, the system can determine whether a failure or excessive load has occurred.

[0030] S602: Dynamic adjustment: Adjust computing power scheduling in a timely manner based on monitoring results. If it is found that the load on a node is too high, it may be necessary to migrate some tasks to other nodes. If the execution of the task does not meet expectations, it may be necessary to re-evaluate the computing power requirements of the task and adjust resource allocation.

[0031] Preferably, the feedback and optimization specifically include:

[0032] S701: Collect feedback information: Collect task execution results and related data, such as the actual task running time and actual resource consumption, and compare them with previous evaluation and prediction results to analyze any differences and problems.

[0033] S702: Optimize models and strategies: Based on feedback information, optimize and improve the resource model, task model, and scheduling strategy. If a scheduling strategy is found to be ineffective in actual application, relevant parameters can be adjusted or new strategies can be adopted to continuously improve the efficiency and performance of computing power scheduling.

[0034] Technical effects and advantages of the present invention:

[0035] 1. This invention achieves precise allocation of computing resources through comprehensive modeling and dynamic computing power evaluation of computing nodes and tasks, combined with a variety of intelligent scheduling strategies, effectively avoiding resource waste and node load imbalance, and significantly improving the overall resource utilization in a distributed computing environment.

[0036] 2. The present invention rationally selects execution nodes and optimizes resource allocation based on the characteristics and requirements of the task, while taking into account task relevance to reduce data transmission overhead, which can speed up task execution, shorten task completion time, and improve the system's processing capability and response speed. Efficient computing power scheduling reduces redundant configuration of hardware resources and unnecessary energy consumption, reduces the construction and operation costs of distributed computing systems, and improves the economic benefits and competitiveness of enterprises.

[0037] 3. Through real-time monitoring and dynamic adjustment mechanisms, the present invention enables the system to respond promptly to situations such as computing node failures, load changes, and task execution anomalies, thereby ensuring the stable operation of the system. Moreover, through the feedback optimization mechanism, the system continuously optimizes the scheduling strategies and models by learning and analyzing historical data, enabling the system to better adapt to complex and changing distributed computing environments and diverse task requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram of the overall structure flow of the present invention. DETAILED DESCRIPTION

[0039] The present invention provides a method for intelligent scheduling of computing power in a distributed computing environment. Figure 1 As shown, the specific steps are:

[0040] S1: Collect resource information data, which is used to collect hardware information, operating status information of computing nodes, and attribute and requirement information of tasks;

[0041] Specifically, resource information data includes basic hardware parameters such as CPU model, number of cores, memory capacity, storage capacity, and network bandwidth. The node's current operating status information includes real-time load data such as CPU usage, memory occupancy, task queue length, disk I / O rate, and network transmission rate.

[0042] Task attributes and requirement information specifically include task type, such as scientific computing, data processing, machine learning training, etc., task priority, which is set by the user or automatically generated based on the importance and urgency of the task, and required computing resource parameters, including the estimated number of CPU cores required, memory requirements, storage requirements, data size, and estimated running time.

[0043] S2: Build resource models and task models for computing nodes, and transform resource and task information into mathematical models that can be calculated and analyzed;

[0044] The specific steps include:

[0045] S201: Resource model construction: Based on the collected node information, a resource model is established to describe the resources of each computing node as a multi-dimensional vector to facilitate subsequent calculations and comparisons;

[0046] S202: Task model construction: Based on the computing requirements of the task, the task is also described as a multi-dimensional vector, corresponding to the resource model.

[0047] S3: Evaluate the available computing power of computing nodes and the computing power requirements of tasks to provide a basis for scheduling decisions;

[0048] The specific steps include:

[0049] S301: Node computing power evaluation: Based on the resource model, the available computing power of each node is evaluated by comprehensively considering the node's hardware performance and current load. For example, the usage of resources such as CPU and memory is quantified through a specific algorithm to obtain a value that reflects the node's current actual computing power.

[0050] S302: Task computing power requirement assessment: Determine the computing power required for the task based on the task model. You can estimate the running time and resource consumption of the task under different resource configurations based on factors such as the task type and scale, combined with historical data or empirical formulas, to determine the required computing power.

[0051] S4: Develop various scheduling strategies such as priority strategy, load balancing strategy, and task correlation strategy based on the characteristics of tasks and resources;

[0052] The specific steps include:

[0053] S401: Based on task priority: Set priorities for different tasks based on factors such as the importance and urgency of the tasks, and prioritize computing resources for high-priority tasks.

[0054] S402: Based on resource utilization: tasks are assigned to nodes with lower resource utilization to balance the load of the entire distributed computing environment and improve overall resource utilization;

[0055] S403: Based on task relevance: If there is relevance between tasks, such as data dependency, try to assign related tasks to the same node or nodes that are close to each other in the network topology to reduce data transmission overhead.

[0056] S5: Computing power allocation and scheduling, using optimization algorithms to perform computing power allocation and task scheduling operations, and reasonably assigning tasks to computing nodes;

[0057] The specific steps include:

[0058] S501: Select nodes according to the scheduling policy: According to the established scheduling policy, the most suitable node is selected from the available computing nodes to execute the task. Various optimization algorithms, such as greedy algorithms and genetic algorithms, can be used to find the optimal allocation solution.

[0059] S502: Allocate computing resources: Allocate tasks to selected nodes and allocate corresponding computing resources to them. Rationally divide and allocate resources on the nodes to ensure that the tasks can run efficiently on the nodes.

[0060] S6: Monitor task execution and node operation status in real time, and make dynamic adjustments based on the monitoring results to ensure stable system operation;

[0061] The specific steps include:

[0062] S601: Monitoring task execution: During task execution, real-time monitoring of the task's running status, including task progress and resource usage, is performed. By monitoring the status of computing nodes, the system can determine whether a failure or excessive load has occurred.

[0063] S602: Dynamic Adjustment: Adjust computing power scheduling in a timely manner based on monitoring results. For example, if the load on a certain node is found to be too high, some tasks may need to be migrated to other nodes. If the execution of a task does not meet expectations, it may be necessary to re-evaluate the computing power requirements of the task and adjust resource allocation.

[0064] S7: Feedback and optimization: collect task execution result data, optimize and improve the system model and scheduling strategy, and improve system performance;

[0065] The specific steps include:

[0066] S701: Collect feedback information: Collect task execution results and related data, such as the actual task running time and actual resource consumption, and compare them with previous evaluation and prediction results to analyze any differences and problems.

[0067] S702: Optimize models and strategies: Based on feedback information, optimize and improve the resource model, task model, and scheduling strategy. If a scheduling strategy is found to be ineffective in actual application, relevant parameters can be adjusted or new strategies can be adopted to continuously improve the efficiency and performance of computing power scheduling.

[0068] It will be understood that the present invention is described by way of some embodiments, and it will be appreciated by those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are intended to be protected by the present invention.

Claims

1. A method for intelligently scheduling computing power in a distributed computing environment, characterized by: The specific steps of the computing power intelligent scheduling method are: S1: Collect resource information data, which is used to collect hardware information, operating status information of computing nodes, and attribute and requirement information of tasks; S2: Build resource models and task models for computing nodes, and transform resource and task information into mathematical models that can be calculated and analyzed; S3: Evaluate the available computing power of computing nodes and the computing power requirements of tasks to provide a basis for scheduling decisions; S4: Develop various scheduling strategies such as priority strategy, load balancing strategy, and task correlation strategy based on the characteristics of tasks and resources; S5: Computing power allocation and scheduling, using optimization algorithms to perform computing power allocation and task scheduling operations, and reasonably assigning tasks to computing nodes; S6: Monitor task execution and node operation status in real time, and make dynamic adjustments based on the monitoring results to ensure stable system operation; S7: Feedback and optimization, collect task execution result data, optimize and improve the system model and scheduling strategy, and improve system performance.

2. The method for intelligently scheduling computing power in a distributed computing environment according to claim 1, wherein: The resource information data specifically includes basic hardware parameters such as CPU model, number of cores, memory capacity, storage capacity, network bandwidth, etc. The current operating status information of the node includes real-time load data such as CPU usage, memory occupancy, task queue length, disk I / O rate, network transmission rate, etc.; The task attributes and requirement information specifically include task type, task priority, required computing resource parameters, data size, and estimated running time.

3. The method for intelligently scheduling computing power in a distributed computing environment according to claim 1, wherein: The resource model and task model of the computing node are constructed as follows: S201: Resource model construction: Based on the collected node information, a resource model is established to describe the resources of each computing node as a multi-dimensional vector to facilitate subsequent calculations and comparisons; S202: Task model construction: Based on the computing requirements of the task, the task is also described as a multi-dimensional vector, corresponding to the resource model.

4. The method for intelligently scheduling computing power in a distributed computing environment according to claim 1, wherein: The specific steps of computing power evaluation include: S301: Node computing power evaluation: Based on the resource model, the available computing power of each node is evaluated by comprehensively considering the node's hardware performance and current load. S302: Task computing power requirement assessment: Determine the computing power required for the task based on the task model. You can estimate the running time and resource consumption of the task under different resource configurations based on factors such as the task type and scale, combined with historical data or empirical formulas, to determine the required computing power.

5. The method for intelligently scheduling computing power in a distributed computing environment according to claim 1, wherein: The formulation of the scheduling strategy specifically includes: S401: Based on task priority: Set priorities for different tasks based on factors such as the importance and urgency of the tasks, and prioritize computing resources for high-priority tasks. S402: Based on resource utilization: tasks are assigned to nodes with lower resource utilization to balance the load of the entire distributed computing environment and improve overall resource utilization; S403: Based on task relevance: If there is relevance between tasks, such as data dependency, try to assign related tasks to the same node or nodes that are close to each other in the network topology to reduce data transmission overhead.

6. The method for intelligently scheduling computing power in a distributed computing environment according to claim 1, wherein: The computing power allocation and scheduling specifically include: S501: Select nodes according to the scheduling policy: According to the established scheduling policy, the most suitable node is selected from the available computing nodes to execute the task. Various optimization algorithms, such as greedy algorithms and genetic algorithms, can be used to find the optimal allocation solution. S502: Allocate computing resources: Allocate tasks to selected nodes and allocate corresponding computing resources to them. Rationally divide and allocate resources on the nodes to ensure that the tasks can run efficiently on the nodes.

7. The method for intelligently scheduling computing power in a distributed computing environment according to claim 1, wherein: The real-time monitoring and adjustment specifically include: S601: Monitoring task execution: During task execution, real-time monitoring of the task's running status, including task progress and resource usage, is performed. By monitoring the status of computing nodes, the system can determine whether a failure or excessive load has occurred. S602: Dynamic adjustment: Adjust computing power scheduling in a timely manner based on monitoring results. If it is found that the load on a node is too high, it may be necessary to migrate some tasks to other nodes. If the execution of the task does not meet expectations, it may be necessary to re-evaluate the computing power requirements of the task and adjust resource allocation.

8. The method for intelligently scheduling computing power in a distributed computing environment according to claim 1, wherein: The feedback and optimization specifically include: S701: Collect feedback information: Collect task execution results and related data, such as the actual task running time and actual resource consumption, and compare them with previous evaluation and prediction results to analyze any differences and problems. S702: Optimize models and strategies: Based on feedback information, optimize and improve the resource model, task model, and scheduling strategy. If a scheduling strategy is found to be ineffective in actual application, relevant parameters can be adjusted or new strategies can be adopted to continuously improve the efficiency and performance of computing power scheduling.

Images