Open distributed system resource management and task scheduling system and method

Abstract

The invention discloses an open distributed system resource management and task scheduling system and method, and relates to the technical field of task scheduling, and the method comprises the steps: calculating operation performance coefficients of different types of subtasks operated by a same node, and outputting an affinity subtask type sorting sequence number of each node after sorting from small to large; calculating the node matching degree of each subtask, sorting from large to small, outputting the node matching degree sorting sequence number of each subtask, multiplying the node matching degree sorting sequence number of each subtask by the affinity processing subtask type sorting sequence number of each node to obtain a comprehensive affinity coefficient, and outputting the node and the subtask corresponding to the minimum comprehensive affinity coefficient to perform execution distribution. All the sub-tasks are executed and allocated; task allocation is dynamically adjusted through the affinity coefficient, overload of some nodes and idling of other nodes are prevented, and the overall stability and throughput of the system are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of task scheduling, and in particular to an open distributed system resource management and task scheduling system and method. Background Art

[0002] With the rapid development of cloud computing, edge computing, big data, and artificial intelligence, distributed systems have become the core infrastructure supporting modern computing. Open distributed systems (such as Kubernetes, Apache Mesos, and Hadoop / YARN) are widely used in scenarios such as large-scale data processing, microservice deployment, and real-time computing due to their high scalability, fault tolerance, and resource sharing capabilities. However, as system scale expands and computing requirements diversify, resource management and task scheduling face numerous challenges.

[0003] In the Chinese invention application with application publication number CN105893157A, an open distributed system resource management and task scheduling system and method are disclosed. It combines a central scheduling function with a resource strategy optimization scheduler, a resource manager, and multiple node managers to simplify the central scheduling function. It solves the problem that resource management and scheduling systems in the existing technology are difficult to directly apply to manage and execute heterogeneous tasks in a heterogeneous hardware resource environment. It can provide a framework for managing heterogeneous hardware resources and scheduling the execution of heterogeneous tasks, and can support dynamic changes in hardware resources and scheduling management of different types of tasks.

[0004] In the above invention application, the resource request sent by the task manager is received, and the hardware status of the resource manager is called. The task manager uses hardware status information to optimize hardware resource scheduling, establishes allocation relationships between multiple subtasks and hardware resources, and sends these allocation relationships to the task manager. However, optimizing hardware resource scheduling based solely on hardware status information without considering the compatibility between tasks and nodes can result in tasks being assigned to incompatible nodes, extending execution time or increasing resource contention. For example, scheduling a deep learning task to a node with high memory but low GPU computing power can significantly reduce computing efficiency.

[0005] To this end, the present invention provides an open distributed system resource management and task scheduling system and method. Summary of the Invention

[0006] (1) Technical problems solved In view of the shortcomings of the existing technology, the present invention provides an open distributed system resource management and task scheduling system and method. The present invention obtains the real-time CPU utilization, real-time memory utilization and real-time bandwidth utilization of each node, and combines the historical average CPU utilization of the current subtask type at different nodes. , average memory usage and average bandwidth usage , calculate the node matching degree of each subtask , sort from large to small, output the node matching degree sorting number of each subtask, multiply it by the affinity processing subtask type sorting number of each node to obtain the comprehensive affinity coefficient, output the node and subtask corresponding to the smallest comprehensive affinity coefficient for execution allocation, until all subtasks are executed and allocated, dynamically adjust task allocation through affinity coefficient to prevent some nodes from being overloaded while other nodes are idle, improve the overall stability and throughput of the system, and thus solve the technical problems recorded in the background technology.

[0007] (2) Technical solution To achieve the above objectives, the present invention is implemented through the following technical solutions: an open distributed system resource management and task scheduling method, comprising the following steps: Obtain hardware resource data, software environment data, and network condition data for each node, and obtain real-time status data for each node through system monitoring tools. Users submit tasks through the scheduling center, which automatically decomposes global tasks into subtask units and marks the subtask types; Obtain each subtask type, extract the average processing time, average CPU utilization, average memory usage, and average bandwidth usage of historical subtasks on different nodes from the node's historical operation log, and calculate the performance coefficient of different types of subtasks running on the same node. , sort from small to large and output the sorting number of the affinity subtask type of each node; Get the real-time CPU utilization, real-time memory utilization, and real-time bandwidth utilization of each node, combined with the historical average CPU utilization of the current subtask type on different nodes , average memory usage and average bandwidth usage , calculate the node matching degree of each subtask , sort from large to small, output the node matching degree sorting number of each subtask, multiply it by the affinity processing subtask type sorting number of each node to get the comprehensive affinity coefficient, output the node and subtask corresponding to the smallest comprehensive affinity coefficient for execution allocation, until all subtasks are executed and allocated; Get the real-time CPU remaining rate of each node , sum up to obtain the real-time CPU surplus rate of the open distributed system. If the real-time CPU surplus rate of the open distributed system is greater than 1, round it down and record it as the number of restable nodes. Based on the number of restable nodes, shut down the number of nodes with the lowest CPU performance ranking.

[0008] Furthermore, each node (physical machine, virtual machine, container, etc.) reports its own resources through the heartbeat mechanism or service registration center (such as Consul, ZooKeeper) to obtain each node's hardware resource data, software environment data, and network condition data.

[0009] Node hardware resource data includes CPU: number of cores, architecture (e.g., x86 / ARM), and hyperthreading support. GPU: model (e.g., NVIDIA A100), video memory size, and CUDA version. Memory: capacity (e.g., 64GB / 128GB), type (DDR4 / DDR5), and bandwidth. Storage: type (SSD / HDD / NVMe), IOPS, and throughput.

[0010] Node software environment data includes the operating system version (e.g., Ubuntu 22.04) and kernel parameters. Dependent libraries include CUDA, TensorFlow, and PyTorch versions. Services include pre-installed databases (MySQL / Redis) and message queues (Kafka).

[0011] Node network condition data includes bandwidth: the intranet bandwidth between nodes (e.g., 10Gbps / 25Gbps), latency: the ping value between nodes (e.g., <1ms), and number of connections: the maximum number of concurrent connections supported.

[0012] Furthermore, the CPU utilization, memory utilization, bandwidth utilization, throughput, network latency, fault status, etc. of each node can be obtained through system monitoring tools.

[0013] Furthermore, users submit tasks through the scheduling center, define task types (such as timed and dependent tasks), and use tools (such as Apache Airflow and Kubernetes) to automatically decompose global tasks into subtask units and mark the subtask types.

[0014] Subtask types include computing tasks (data cleaning, algorithm training), storage tasks (database writing, file backup) and communication tasks (API calls, message queue transmission).

[0015] Furthermore, the subtask types are obtained and the average processing time of historical subtasks of different nodes is extracted from the node historical operation log. , average CPU utilization , average memory usage and average bandwidth usage , calculate the performance coefficient of different types of subtasks running on the same node :

[0016] Wherein, i represents the sequential number of each node of the open distributed system, i=1, 2, ..., n, n represents the total number of nodes in the open distributed system, j represents the sequential number of the subtask type, j=1, 2, 3.

[0017] Furthermore, the performance coefficients of different types of subtasks running on the same node are calculated. Sort from small to large and output the sorting number of the affinity subtask type of each node. The type of subtask with a smaller sorting number has the strongest affinity with the node.

[0018] Furthermore, the real-time CPU utilization, real-time memory utilization, and real-time bandwidth utilization of each node are obtained and converted into real-time CPU remaining rate. , Real-time memory remaining and real-time bandwidth remaining :

[0019] Furthermore, the historical average CPU utilization of different nodes in combination with the current subtask type , average memory usage and average bandwidth usage , calculate the node matching degree of each subtask :

[0020] Furthermore, the node matching degree of each subtask Sort from large to small and output the node matching degree sorting number of each subtask. The node with the smaller sorting number has the strongest affinity for the subtask.

[0021] Furthermore, the affinity processing subtask type sorting number of each node and the node matching degree sorting number of each subtask are obtained, and the comprehensive affinity coefficient is obtained after multiplication. The node and subtask corresponding to the smallest comprehensive affinity coefficient are output for execution allocation. Then, the comprehensive affinity coefficients of the remaining subtasks and nodes are re-analyzed for execution allocation until all subtasks are executed and allocated.

[0022] An open distributed system resource management and task scheduling system, comprising: The initial data collation module obtains hardware resource data, software environment data, and network condition data of each node, and obtains real-time status data of each node through system monitoring tools. Users submit tasks through the scheduling center, which automatically decomposes global tasks into subtask units and marks the subtask type; The node affinity analysis module obtains the types of each subtask, extracts the average processing time, average CPU utilization, average memory usage, and average bandwidth usage of historical subtasks on different nodes from the node's historical operation log, and calculates the performance coefficient of different types of subtasks running on the same node. , sort from small to large and output the sorting number of the affinity subtask type of each node; The task matching module obtains the real-time CPU utilization, real-time memory utilization, and real-time bandwidth utilization of each node, combined with the historical average CPU utilization of the current subtask type on different nodes , average memory usage and average bandwidth usage , calculate the node matching degree of each subtask , sort from large to small, output the node matching degree sorting number of each subtask, multiply it by the affinity processing subtask type sorting number of each node to get the comprehensive affinity coefficient, output the node and subtask corresponding to the smallest comprehensive affinity coefficient for execution allocation, until all subtasks are executed and allocated; System redundancy analysis module to obtain the real-time CPU surplus rate of each node , sum up to obtain the real-time CPU surplus rate of the open distributed system. If the real-time CPU surplus rate of the open distributed system is greater than 1, round it down and record it as the number of restable nodes. Based on the number of restable nodes, shut down the number of nodes with the lowest CPU performance ranking.

[0023] (3) Beneficial effects The present invention provides an open distributed system resource management and task scheduling system and method, which has the following beneficial effects: 1. Obtain hardware resource data, software environment data, and network condition data for each node, and obtain real-time status data for each node through system monitoring tools. Users submit tasks through the scheduling center, which automatically decomposes global tasks into subtask units and marks the subtask type. Automatically mark subtask types (such as computing type and storage type) to match the most suitable node environment, reduce the risk of manual configuration errors, and reduce manual intervention and operation and maintenance complexity through automated processes. At the same time, reduce hardware redundancy requirements through precise resource utilization.

[0024] 2. Obtain each subtask type, extract the average processing time, average CPU utilization, average memory usage, and average bandwidth usage of historical subtasks on different nodes from the node's historical operation log, and calculate the performance coefficient of different types of subtasks running on the same node. , after sorting from small to large, output the affinity subtask type sorting number of each node. By analyzing historical data, the average processing time, resource usage and other indicators of different subtasks on the node are clarified to achieve the optimal matching of tasks and nodes and reduce resource idleness or overload.

[0025] 3. Obtain the real-time CPU utilization, real-time memory utilization, and real-time bandwidth utilization of each node, combined with the historical average CPU utilization of the current subtask type on different nodes , average memory usage and average bandwidth usage , calculate the node matching degree of each subtask , sort from large to small, output the node matching degree sorting number of each subtask, multiply it by the affinity processing subtask type sorting number of each node to get the comprehensive affinity coefficient, output the node and subtask corresponding to the smallest comprehensive affinity coefficient for execution allocation, until all subtasks are executed and allocated, dynamically adjust task allocation through affinity coefficient to prevent some nodes from being overloaded while other nodes are idle, and improve the overall stability and throughput of the system.

[0026] 4. Get the real-time CPU remaining rate of each node , sum up to obtain the real-time CPU surplus rate of the open distributed system. If the real-time CPU surplus rate of the open distributed system is greater than 1, it is rounded down and recorded as the number of resting nodes. According to the number of resting nodes, the number of nodes with lower CPU performance ranking is shut down. Shutting down the nodes with lower performance ranking can reduce unnecessary power consumption, directly reduce operating costs, reduce the running time of low-load nodes, and reduce the speed of hardware aging. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A schematic diagram of a flow chart of an open distributed system resource management and task scheduling method of the present invention; Figure 2 This is a structural diagram of an open distributed system resource management and task scheduling system of the present invention. DETAILED DESCRIPTION

[0028] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0029] See also Figure 1 The present invention provides an open distributed system resource management and task scheduling method, comprising the following steps: Step 1: Obtain hardware resource data, software environment data, and network condition data for each node, and obtain real-time status data for each node through system monitoring tools. Users submit tasks through the scheduling center, which automatically decomposes global tasks into subtask units and marks the subtask types.

[0030] The step 1 includes the following: Step 101: Each node (physical machine, virtual machine, container, etc.) reports its own resources through the heartbeat mechanism or service registration center (such as Consul, ZooKeeper) to obtain hardware resource data, software environment data, and network condition data of each node.

[0031] Node hardware resource data includes CPU: number of cores, architecture (e.g., x86 / ARM), and hyperthreading support. GPU: model (e.g., NVIDIA A100), video memory size, and CUDA version. Memory: capacity (e.g., 64GB / 128GB), type (DDR4 / DDR5), and bandwidth. Storage: type (SSD / HDD / NVMe), IOPS, and throughput.

[0032] Node software environment data includes the operating system version (e.g., Ubuntu 22.04) and kernel parameters. Dependent libraries include CUDA, TensorFlow, and PyTorch versions. Services include pre-installed databases (MySQL / Redis) and message queues (Kafka).

[0033] Node network condition data includes bandwidth: the intranet bandwidth between nodes (e.g., 10Gbps / 25Gbps), latency: the ping value between nodes (e.g., <1ms), and number of connections: the maximum number of concurrent connections supported.

[0034] Step 102: The CPU utilization, memory utilization, bandwidth utilization, throughput, network delay, fault status, etc. of each node are obtained through the system monitoring tool.

[0035] Step 103: The user submits a task through the scheduling center, defines the task type (such as timed or dependent tasks), and uses tools (such as Apache Airflow or Kubernetes) to automatically decompose the global task into subtask units and mark the subtask type.

[0036] Subtask types include computing tasks (data cleaning, algorithm training), storage tasks (database writing, file backup) and communication tasks (API calls, message queue transmission).

[0037] When using, combine the contents in steps 101 to 103: Obtain hardware resource data, software environment data, and network condition data for each node, and obtain real-time status data for each node through system monitoring tools. Users submit tasks through the scheduling center, which automatically decomposes global tasks into subtask units and marks the subtask types. It automatically marks subtask types (such as computing type and storage type) to match the most suitable node environment, reduce the risk of manual configuration errors, and reduce manual intervention and operation and maintenance complexity through automated processes. At the same time, it reduces hardware redundancy requirements through precise resource utilization.

[0038] Step 2: Obtain each subtask type, extract the average processing time, average CPU utilization, average memory usage, and average bandwidth usage of historical subtasks on different nodes from the node's historical operation log, and calculate the performance coefficient of different types of subtasks running on the same node. , sort from small to large and output the sorting sequence number of the affinity subtask type of each node.

[0039] The second step includes the following: Step 201: Obtain each subtask type and extract the average processing time of historical subtasks of different nodes from the node historical operation log. , average CPU utilization , average memory usage and average bandwidth usage , calculate the performance coefficient of different types of subtasks running on the same node :

[0040] Wherein, i represents the sequential number of each node of the open distributed system, i=1, 2, ..., n, n represents the total number of nodes in the open distributed system, j represents the sequential number of the subtask type, j=1, 2, 3.

[0041] Step 202: The performance coefficients of different subtasks running on the same node are calculated. Sort from small to large and output the sorting number of the affinity subtask type of each node. The type of subtask with a smaller sorting number has the strongest affinity with the node.

[0042] When using, combine the contents in steps 201 and 202: Obtain each subtask type, extract the average processing time, average CPU utilization, average memory usage, and average bandwidth usage of historical subtasks on different nodes from the node's historical operation log, and calculate the performance coefficient of different types of subtasks running on the same node. , after sorting from small to large, output the affinity subtask type sorting number of each node. By analyzing historical data, the average processing time, resource usage and other indicators of different subtasks on the node are clarified to achieve the optimal matching of tasks and nodes and reduce resource idleness or overload.

[0043] Step 3: Get the real-time CPU utilization, real-time memory utilization, and real-time bandwidth utilization of each node, combined with the historical average CPU utilization of the current subtask type on different nodes , average memory usage and average bandwidth usage , calculate the node matching degree of each subtask , sort from large to small, output the node matching degree sorting number of each subtask, multiply it by the affinity processing subtask type sorting number of each node to obtain the comprehensive affinity coefficient, output the node and subtask corresponding to the smallest comprehensive affinity coefficient for execution allocation, until all subtasks are executed and allocated.

[0044] The step three includes the following: Step 301: Get the real-time CPU utilization, real-time memory utilization and real-time bandwidth utilization of each node and convert them into real-time CPU remaining rate. , Real-time memory remaining and real-time bandwidth remaining :

[0045] Step 302: Combine the historical average CPU utilization of different nodes based on the current subtask type , average memory usage and average bandwidth usage , calculate the node matching degree of each subtask :

[0046] The node matching degree of each subtask Sort from large to small and output the node matching degree sorting number of each subtask. The node with the smaller sorting number has the strongest affinity for the subtask.

[0047] Step 303: Obtain the affinity processing subtask type sorting number of each node and the node matching degree sorting number of each subtask, multiply them to obtain the comprehensive affinity coefficient, output the node and subtask corresponding to the smallest comprehensive affinity coefficient for execution allocation, and then re-analyze the comprehensive affinity coefficients of the remaining subtasks and nodes for execution allocation until all subtasks are executed and allocated.

[0048] When using, combine the contents in steps 301 to 303: Get the real-time CPU utilization, real-time memory utilization, and real-time bandwidth utilization of each node, combined with the historical average CPU utilization of the current subtask type on different nodes , average memory usage and average bandwidth usage , calculate the node matching degree of each subtask , sort from large to small, output the node matching degree sorting number of each subtask, multiply it by the affinity processing subtask type sorting number of each node to get the comprehensive affinity coefficient, output the node and subtask corresponding to the smallest comprehensive affinity coefficient for execution allocation, until all subtasks are executed and allocated, dynamically adjust task allocation through affinity coefficient to prevent some nodes from being overloaded while other nodes are idle, and improve the overall stability and throughput of the system.

[0049] Step 4: Get the real-time CPU remaining rate of each node , sum up to obtain the real-time CPU surplus rate of the open distributed system. If the real-time CPU surplus rate of the open distributed system is greater than 1, round it down and record it as the number of restable nodes. Based on the number of restable nodes, shut down the number of nodes with the lowest CPU performance ranking.

[0050] The step 4 includes the following contents: Step 401: Get the real-time CPU remaining rate of each node , sum up to obtain the real-time CPU surplus rate of the open distributed system. If the real-time CPU surplus rate of the open distributed system is greater than 1, it is rounded down and recorded as the number of restable nodes.

[0051] Step 402: shut down the number of nodes with the lowest CPU performance ranking according to the number of restable nodes.

[0052] Among them, CPU performance ranking is achieved through CPU performance testing tools.

[0053] When using, combine the contents in steps 401 and 402: Get the real-time CPU remaining rate of each node , sum up to obtain the real-time CPU surplus rate of the open distributed system. If the real-time CPU surplus rate of the open distributed system is greater than 1, it is rounded down and recorded as the number of resting nodes. According to the number of resting nodes, the number of nodes with lower CPU performance ranking is shut down. Shutting down the nodes with lower performance ranking can reduce unnecessary power consumption, directly reduce operating costs, reduce the running time of low-load nodes, and reduce the speed of hardware aging.

[0054] See also Figure 2 The present invention provides an open distributed system resource management and task scheduling system, comprising: The initial data collation module obtains the hardware resource data, software environment data and network condition data of each node, and obtains the real-time status data of each node through the system monitoring tool. Users submit tasks through the scheduling center, which automatically decomposes the global task into subtask units and marks the subtask type.

[0055] The node affinity analysis module obtains the subtask type, extracts the average processing time, average CPU utilization, average memory usage, and average bandwidth usage of historical subtasks on different nodes from the node's historical operation log, and calculates the performance coefficient of different types of subtasks running on the same node. , sort from small to large and output the sorting sequence number of the affinity subtask type of each node.

[0056] The task matching module obtains the real-time CPU utilization, real-time memory utilization, and real-time bandwidth utilization of each node, combined with the historical average CPU utilization of the current subtask type on different nodes , average memory usage and average bandwidth usage , calculate the node matching degree of each subtask , sort from large to small, output the node matching degree sorting number of each subtask, multiply it by the affinity processing subtask type sorting number of each node to obtain the comprehensive affinity coefficient, output the node and subtask corresponding to the smallest comprehensive affinity coefficient for execution allocation, until all subtasks are executed and allocated.

[0057] System redundancy analysis module to obtain the real-time CPU surplus rate of each node , sum up to obtain the real-time CPU surplus rate of the open distributed system. If the real-time CPU surplus rate of the open distributed system is greater than 1, round it down and record it as the number of restable nodes. Based on the number of restable nodes, shut down the number of nodes with the lowest CPU performance ranking.

[0058] The present invention provides another specific embodiment: Get the real-time CPU surplus rate of each node as 0.2, 0.5, 0.6, 0.8, and 0.4, and sum them to obtain the real-time CPU surplus rate of the open distributed system: 0.2 + 0.5 + 0.6 + 0.8 + 0.4 = 2.5.

[0059] If the sum of the real-time CPU surplus rates of the open distributed system is greater than 1, the value 2 is rounded down to the nearest integer and recorded as the number of resting nodes. The two nodes with the lowest CPU performance ranking are shut down.

[0060] The above embodiments can be implemented in whole or in part by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. Those skilled in the art will appreciate that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution.

[0061] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0062] The above is only a specific implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

Claims

1. A method for resource management and task scheduling in an open distributed system, characterized by: The steps include: Obtain hardware resource data, software environment data, and network condition data for each node, and obtain real-time status data for each node through system monitoring tools. Users submit tasks through the scheduling center, which automatically decomposes global tasks into subtask units and marks the subtask types; Obtain each subtask type, extract the average processing time, average CPU utilization, average memory usage, and average bandwidth usage of historical subtasks on different nodes from the node's historical operation log, and calculate the performance coefficient of different types of subtasks running on the same node. , sort from small to large and output the sorting number of the affinity subtask type of each node; Get the real-time CPU utilization, real-time memory utilization, and real-time bandwidth utilization of each node, combined with the historical average CPU utilization of the current subtask type on different nodes , average memory usage and average bandwidth usage , calculate the node matching degree of each subtask , sort from large to small, output the node matching degree sorting number of each subtask, multiply it by the affinity processing subtask type sorting number of each node to get the comprehensive affinity coefficient, output the node and subtask corresponding to the smallest comprehensive affinity coefficient for execution allocation, until all subtasks are executed and allocated; Get the real-time CPU remaining rate of each node , sum up to obtain the real-time CPU surplus rate of the open distributed system. If the real-time CPU surplus rate of the open distributed system is greater than 1, round it down and record it as the number of restable nodes. Based on the number of restable nodes, shut down the number of nodes with the lowest CPU performance ranking.

2. The open distributed system resource management and task scheduling method according to claim 1, characterized in that: Get each subtask type and extract the average processing time of historical subtasks of different nodes from the node historical operation log , average CPU utilization , average memory usage and average bandwidth usage , calculate the performance coefficient of different types of subtasks running on the same node : Wherein, i represents the sequential number of each node of the open distributed system, i=1, 2, ..., n, n represents the total number of nodes in the open distributed system, j represents the sequential number of the subtask type, j=1, 2, 3.

3. The open distributed system resource management and task scheduling method according to claim 1, characterized in that: The performance coefficient of running different types of subtasks on the same node Sort from small to large and output the sorting number of the affinity subtask type of each node. The type of subtask with a smaller sorting number has the strongest affinity with the node.

4. The open distributed system resource management and task scheduling method according to claim 1, characterized in that: Get the real-time CPU utilization, real-time memory utilization, and real-time bandwidth utilization of each node and convert them into real-time CPU remaining rate , Real-time memory remaining and real-time bandwidth remaining : 。 5. The open distributed system resource management and task scheduling method according to claim 1, characterized in that: Combined with the historical average CPU utilization of the current subtask type on different nodes , average memory usage and average bandwidth usage , calculate the node matching degree of each subtask : 。 6. The open distributed system resource management and task scheduling method according to claim 1, characterized in that: The node matching degree of each subtask Sort from large to small and output the node matching degree sorting number of each subtask. The node with the smaller sorting number has the strongest affinity for the subtask.

7. The open distributed system resource management and task scheduling method according to claim 1, characterized in that: Obtain the affinity processing subtask type sorting number of each node and the node matching degree sorting number of each subtask, multiply them to obtain the comprehensive affinity coefficient, output the node and subtask corresponding to the smallest comprehensive affinity coefficient for execution allocation, and then re-analyze the comprehensive affinity coefficients of the remaining subtasks and nodes for execution allocation until all subtasks are executed and allocated.

8. An open distributed system resource management and task scheduling system, used to implement the method according to any one of claims 1 to 7, characterized in that: include: The initial data collation module obtains hardware resource data, software environment data, and network condition data of each node, and obtains real-time status data of each node through system monitoring tools. Users submit tasks through the scheduling center, which automatically decomposes global tasks into subtask units and marks the subtask type; The node affinity analysis module obtains the types of each subtask, extracts the average processing time, average CPU utilization, average memory usage, and average bandwidth usage of historical subtasks on different nodes from the node's historical operation log, and calculates the performance coefficient of different types of subtasks running on the same node. , sort from small to large and output the sorting number of the affinity subtask type of each node; The task matching module obtains the real-time CPU utilization, real-time memory utilization, and real-time bandwidth utilization of each node, combined with the historical average CPU utilization of the current subtask type on different nodes , average memory usage and average bandwidth usage , calculate the node matching degree of each subtask , sort from large to small, output the node matching degree sorting number of each subtask, multiply it by the affinity processing subtask type sorting number of each node to get the comprehensive affinity coefficient, output the node and subtask corresponding to the smallest comprehensive affinity coefficient for execution allocation, until all subtasks are executed and allocated; System redundancy analysis module to obtain the real-time CPU surplus rate of each node , sum up to obtain the real-time CPU surplus rate of the open distributed system. If the real-time CPU surplus rate of the open distributed system is greater than 1, round it down and record it as the number of restable nodes. Based on the number of restable nodes, shut down the number of nodes with the lowest CPU performance ranking.

Images