An open distributed system resource management and task scheduling system and method

Abstract

The application discloses an open distributed system resource management and task scheduling system and method, relates to the technical field of task scheduling, and comprises the following steps: calculating the running performance coefficients of different types of subtasks running on the same node, and outputting the affinity subtask type order numbers of each node after being sorted from small to large; calculating the node matching degrees of each subtask, sorting from large to small, and outputting the node matching degree order numbers of each subtask; multiplying the node matching degree order numbers of each subtask by the affinity processing subtask type order numbers of each node to obtain comprehensive affinity coefficients; and outputting the node and subtask corresponding to the smallest comprehensive affinity coefficient for execution allocation until all subtasks are executed and allocated. The task allocation is dynamically adjusted through the affinity coefficients, overloading of some nodes and idling of other nodes are prevented, and the overall stability and throughput of the system are improved.

Description

Technical Field

[0001] This invention relates to the field of task scheduling technology, specifically to an open distributed system resource management and task scheduling system and method. Background Technology

[0002] With the rapid development of cloud computing, edge computing, big data, and artificial intelligence, distributed systems have become the core infrastructure supporting modern computing. Open distributed systems (such as Kubernetes, Apache Mesos, Hadoop / YARN, etc.) are widely used in scenarios such as large-scale data processing, microservice deployment, and real-time computing due to their high scalability, fault tolerance, and resource sharing capabilities. However, as system scale increases and computing demands diversify, resource management and task scheduling face numerous challenges.

[0003] Chinese invention application CN105893157A discloses an open distributed system resource management and task scheduling system and method, which divides the open distributed system resource management and task scheduling system into task management...

[0004] The system includes a processor, a resource policy optimization scheduler, a resource manager, and multiple node managers, simplifying the central scheduling function. It solves the problem that existing resource management and scheduling systems are difficult to apply directly to manage and execute heterogeneous tasks in heterogeneous hardware resource environments. It provides a framework for managing heterogeneous hardware resources and scheduling heterogeneous tasks, supporting dynamic changes in hardware resources and scheduling management of different types of tasks.

[0005] In the above invention applications, the system receives a resource request sent by the task manager and calls the hardware status of the resource manager.

[0006] The system optimizes hardware resource scheduling based on hardware status information, establishes allocation relationships between multiple subtasks and hardware resources, and sends these relationships to the task manager. However, relying solely on hardware status information for optimized hardware resource scheduling without considering task-node compatibility may result in tasks being assigned to mismatched nodes, prolonging execution time or increasing resource contention. For example, scheduling a deep learning task to a node with high memory but low GPU computing power significantly reduces computational efficiency.

[0007] Therefore, the present invention provides an open distributed system resource management and task scheduling system and method. Summary of the Invention

[0008] (a) Technical problems to be solved

[0009] In view of the deficiencies of the prior art, the present application provides an open distributed system resource management and task scheduling system and method , average memory usage , and average bandwidth usage , the node matching degree of each subtask is calculated , sorted from large to small, the node matching degree sorting sequence number of each subtask is output, multiplied by the affinity processing subtask type sorting sequence number of each node to obtain a comprehensive affinity coefficient, the node and the subtask corresponding to the smallest comprehensive affinity coefficient are output for execution allocation, until all subtasks are executed and allocated, the task allocation is dynamically adjusted through the affinity coefficient, the problem of some nodes being overloaded while other nodes being idle is prevented, and the overall stability and throughput of the system are improved, thereby solving the technical problems described in the background art.

[0010] (II) Technical solutions

[0011] To achieve the above object, the present application is implemented by the following technical solutions: an open distributed system resource management and task scheduling method, comprising the following steps:

[0012] Obtain the hardware resource data, software environment data and network condition data of each node, and obtain the real-time state data of each node through a system monitoring tool, a user submits a task through a scheduling center, the global task is automatically decomposed into subtask units, and the subtask type is marked;

[0013] Obtain each subtask type, extract the average processing time, average CPU utilization, average memory usage and average bandwidth usage of the historical type subtask in different nodes from the historical running log of the node, calculate the running performance coefficient of different types of subtasks running on the same node , and output the affinity subtask type sorting sequence number of each node after sorting from small to large;

[0014] Obtain the real-time CPU utilization, real-time memory utilization and real-time bandwidth utilization of each node, combine the historical average CPU utilization , average memory usage and average bandwidth usage of the current subtask type in different nodes, calculate the node matching degree of each subtask , sort from large to small, output the node matching degree sorting sequence number of each subtask, multiply the affinity processing subtask type sorting sequence number of each node to obtain a comprehensive affinity coefficient, output the node and the subtask corresponding to the smallest comprehensive affinity coefficient for execution allocation, until all subtasks are executed and allocated;

[0015] Obtain the real-time CPU remaining rate of each node Sum up to obtain the open distributed system real-time CPU remaining rate and, if the open distributed system real-time CPU remaining rate is greater than 1, then take the integer part and record it as the number of nodes that can rest, and according to the number of nodes that can rest, turn off the corresponding CPU performance ranking nodes.

[0016] Further, each node (physical machine, virtual machine, container, etc.) reports its own resources through a heartbeat mechanism or a service registration center (such as Consul, ZooKeeper) to obtain hardware resource data, software environment data, and network condition data of each node.

[0017] Node hardware resource data includes CPU: core number, architecture (such as x86 / ARM), whether it supports hyper-threading. GPU: model (such as NVIDIA A100), video memory size, CUDA version. Memory: capacity (such as 64GB / 128GB), type (DDR4 / DDR5), bandwidth. Storage: type (SSD / HDD / NVMe), IOPS, throughput.

[0018] Node software environment data includes operating system: version (such as Ubuntu 22.04), kernel parameters. Dependency library: such as CUDA, TensorFlow, PyTorch version. Service: pre-installed database (MySQL / Redis), message queue (Kafka).

[0019] Node network condition data includes bandwidth: intranet bandwidth between nodes (such as 10Gbps / 25Gbps). Delay: inter-node ping value (such as <1ms). Connection number: maximum number of concurrent connections supported.

[0020] Further, through system monitoring tools, obtain the CPU utilization, memory utilization, bandwidth utilization, throughput, network delay, fault status, etc. of each node.

[0021] Further, users submit tasks through the scheduling center, define task types (such as timing, dependent tasks), and use tools (such as Apache Airflow, Kubernetes) to automatically decompose global tasks into sub-task units and mark sub-task types.

[0022] Sub-task types include computing tasks (data cleaning, algorithm training), storage tasks (database writing, file backup), and communication tasks (API calling, message queue transmission).

[0023] Further, obtain each sub-task type, extract the average processing time of historical type sub-tasks on different nodes from the historical running logs of the nodes average CPU utilization average memory usage average bandwidth usage performance coefficient of the same node running different types of sub-tasks :

[0024]

[0025] wherein i represents the sequential number of each node of the open distributed system, i = 1, 2, …, n, n represents the total number of nodes of the open distributed system, and j represents the sequential number of the sub-task type, j = 1, 2, 3.

[0026] Further, the performance coefficients of the same node running different types of sub-tasks are sorted from small to large, and the affinity sub-task type sorting sequence number of each node is output, and the smaller the sorting sequence number, the stronger the affinity of the type sub-task to the node.

[0027] Further, the real-time CPU utilization, real-time memory utilization and real-time bandwidth utilization of each node are obtained, which are converted into real-time CPU remaining rate , real-time memory remaining amount and real-time bandwidth remaining amount :

[0028]

[0029] Further, combined with the historical average CPU utilization , average memory usage and average bandwidth usage of the current sub-task type in different nodes, the node matching degree of each sub-task is calculated :

[0030]

[0031] Further, the node matching degree of each sub-task is sorted from large to small, and the node matching degree sorting sequence number of each sub-task is output, and the smaller the sorting sequence number, the stronger the affinity of the sub-task corresponding to the node.

[0032] Further, the affinity processing sub-task type sorting sequence number of each node and the node matching degree sorting sequence number of each sub-task are obtained, multiplied to obtain a comprehensive affinity coefficient, and the node and sub-task corresponding to the smallest comprehensive affinity coefficient are output for execution allocation, and then the comprehensive affinity coefficients of the remaining sub-tasks and nodes are re-analyzed for execution allocation until all sub-tasks are executed and allocated.

[0033] An open distributed system resource management and task scheduling system comprises:

[0034] An initial data arrangement module acquires hardware resource data, software environment data and network condition data of each node, and obtains real-time state data of each node through a system monitoring tool, a user submits a task through a scheduling center, and automatically decomposes a global task into subtask units and marks subtask types;

[0035] A node affinity analysis module acquires each subtask type, extracts average processing time length, average CPU utilization, average memory usage and average bandwidth usage of historical type subtasks in different nodes from a node historical running log, and calculates running performance coefficients of different types of subtasks running on the same node , and outputs an affinity subtask type sorting serial number of each node after sorting from small to large;

[0036] A task matching module acquires real-time CPU utilization, real-time memory utilization and real-time bandwidth utilization of each node, combines historical average CPU utilization , average memory usage and average bandwidth usage of current subtask types in different nodes, calculates node matching degrees of each subtask , sorts from large to small, and outputs a node matching degree sorting serial number of each subtask; after multiplying the node matching degree sorting serial number of each node and the affinity processing subtask type sorting serial number, a comprehensive affinity coefficient is obtained, a node and a subtask corresponding to the smallest comprehensive affinity coefficient are executed and allocated, and the execution and allocation of all subtasks are completed.

[0037] A system redundancy analysis module acquires real-time CPU residual rates of each node , sums up to obtain real-time CPU residual rates of an open distributed system, and if the real-time CPU residual rates of the open distributed system are greater than 1, the open distributed system is rounded down to be recorded as a restable node number, and according to the restable node number, a corresponding CPU performance sorting rear number of nodes is closed.

[0038] (Three) beneficial effects

[0039] The present application provides an open distributed system resource management and task scheduling system and method, which has the following beneficial effects:

[0040] 1, Obtain the hardware resource data, software environment data and network condition data of each node, and obtain the real-time state data of each node through the system monitoring tool, the user submits the task through the scheduling center, automatically decomposes the global task into subtask units, and marks the subtask type, automatically marks the subtask type (such as computing type, storage type), matches the most suitable node environment, reduces the risk of manual configuration error, reduces the manual intervention of automatic process, reduces the operation and maintenance complexity, and reduces the hardware redundancy demand through accurate resource utilization.

[0041] 2, Obtain each subtask type, extract the average processing time, average CPU utilization, average memory usage and average bandwidth usage of the historical type subtask in different nodes from the node historical running log, calculate the running performance coefficient of different types of subtask running on the same node , sort the affinity subtask type sorting sequence number of each node from small to large, analyze the historical data to determine the average processing time, resource occupation and other indicators of different subtasks on the node, realize the optimal matching of task and node, and reduce resource idling or overload.

[0042] 3, Obtain the real-time CPU utilization, real-time memory utilization and real-time bandwidth utilization of each node, combine the historical average CPU utilization , average memory usage and average bandwidth usage of the current subtask type in different nodes , calculate the node matching degree of each subtask , sort from large to small, output the node matching degree sorting sequence number of each subtask, multiply the affinity processing subtask type sorting sequence number of each node to obtain the comprehensive affinity coefficient, output the node and subtask corresponding to the smallest comprehensive affinity coefficient for execution allocation, until all subtasks are executed and allocated, dynamically adjust the task allocation through the affinity coefficient, prevent some nodes from being overloaded while other nodes are idle, and improve the overall stability and throughput of the system.

[0043] 4, Obtain the real-time CPU remaining rate of each node, sum the real-time CPU remaining rate of the open distributed system to obtain the real-time CPU remaining rate, if the real-time CPU remaining rate of the open distributed system is greater than 1, then take the integer part after rounding down as the number of nodes that can rest, according to the number of nodes that can rest, close the number of nodes with the last performance ranking, closing the nodes with the last performance ranking can reduce unnecessary power consumption, directly reduce the operation cost, reduce the running time of low-load nodes, and reduce the hardware aging speed.

[0044] Figure 1 It is a flowchart of the resource management and task scheduling method of the open distributed system of the application;

[0045] Figure 2 This is a schematic diagram of the structure of an open distributed system resource management and task scheduling system according to the present invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] Please see Figure 1 This invention provides a method for resource management and task scheduling in an open distributed system, comprising the following steps:

[0048] Step 1: Obtain hardware resource data, software environment data, and network condition data for each node, and obtain real-time status data for each node through system monitoring tools. Users submit tasks through the scheduling center, which automatically decomposes global tasks into sub-task units and marks the sub-task types.

[0049] Step one includes the following:

[0050] Step 101: Each node (physical machine, virtual machine, container, etc.) reports its own resources through a heartbeat mechanism or service registry (such as Consul, ZooKeeper) to obtain hardware resource data, software environment data and network condition data of each node.

[0051] Node hardware resource data includes: CPU: number of cores, architecture (e.g., x86 / ARM), and whether it supports hyper-threading; GPU: model (e.g., NVIDIA A100), memory size, and CUDA version; Memory: capacity (e.g., 64GB / 128GB), type (DDR4 / DDR5), and bandwidth; Storage: type (SSD / HDD / NVMe), IOPS, and throughput.

[0052] Node software environment data includes: operating system: version (e.g., Ubuntu 22.04), kernel parameters; dependent libraries: such as CUDA, TensorFlow, and PyTorch versions; services: pre-installed database (MySQL / Redis) and message queue (Kafka).

[0053] Node network condition data includes: Bandwidth: Inter-node intranet bandwidth (e.g., 10Gbps / 25Gbps). Latency: Inter-node ping value (e.g., <1ms). Connections: Maximum supported concurrent connections.

[0054] Step 102, obtain CPU utilization, memory utilization, bandwidth utilization, throughput, network delay, fault status, etc. of each node through system monitoring tools.

[0055] Step 103, user submits tasks through scheduling center, defines task type (such as timing, dependent task), uses tools (such as Apache Airflow, Kubernetes) to automatically decompose global tasks into subtask units, and marks subtask type.

[0056] Subtask type includes computing task (data cleaning, algorithm training), storage task (database writing, file backup) and communication task (API calling, message queue transmission).

[0057] In use, in combination with the contents in steps 101 to 103:

[0058] Obtain each node hardware resource data, software environment data and network condition data, and obtain real-time state data of each node through system monitoring tools, user submits tasks through scheduling center, automatically decomposes global tasks into subtask units, and marks subtask type, automatically marks subtask type (such as computing type, storage type), matches the most suitable node environment, reduces the risk of manual configuration error, reduces the complexity of operation and maintenance through automatic process and reduces the demand for hardware redundancy through precise resource utilization.

[0059] Step two, obtain each subtask type, extract the average processing time, average CPU utilization, average memory usage and average bandwidth usage of historical type subtasks in different nodes from node historical running logs, and calculate the running performance coefficient of running different types of subtasks in the same node Sort the affinity subtask type sorting serial number of each node from small to large.

[0060] The step two includes the following contents:

[0061] Step 201, obtain each subtask type, extract the average processing time , average CPU utilization , average memory usage and average bandwidth usage of historical type subtasks in different nodes from node historical running logs, and calculate the running performance coefficient of running different types of subtasks in the same node :

[0062]

[0063] Where i represents the sequential number of each node in the open distributed system, i=1, 2, ..., n, n represents the total number of nodes in the open distributed system, and j represents the sequential number of the subtask type, j=1, 2, 3.

[0064] Step 202: Compare the performance coefficients of different types of subtasks running on the same node. Sort the nodes from smallest to largest and output the sorting index of the affinity subtask types for each node. The smaller the sorting index, the stronger the affinity between the subtask type and the node.

[0065] When using it, refer to steps 201 and 202:

[0066] Obtain the subtask types, extract the average processing time, average CPU utilization, average memory usage, and average bandwidth usage of historical subtask types across different nodes from the node's historical execution logs, and calculate the performance coefficients for running different types of subtasks on the same node. After sorting from smallest to largest, the system outputs the affinity subtask type sorting number for each node. By analyzing historical data, it clarifies the average processing time, resource consumption, and other indicators of different subtasks on the nodes, achieving optimal matching between tasks and nodes and reducing resource idleness or overload.

[0067] Step 3: Obtain the real-time CPU utilization, real-time memory utilization, and real-time bandwidth utilization of each node, and combine this with the historical average CPU utilization of the current subtask type on different nodes. Average memory usage and average bandwidth usage Calculate the node matching degree for each subtask. Sort the nodes from largest to smallest, output the node matching degree sorting number of each subtask, multiply it by the affinity processing subtask type sorting number of each node to obtain the comprehensive affinity coefficient, output the node and subtask corresponding to the smallest comprehensive affinity coefficient for execution allocation, until all subtasks have been executed and allocated.

[0068] Step three includes the following:

[0069] Step 301: Obtain the real-time CPU utilization, real-time memory utilization, and real-time bandwidth utilization of each node, and convert them into real-time CPU remaining rate. Real-time remaining memory and real-time bandwidth remaining :

[0070]

[0071] Step 302: Combine the historical average CPU utilization of the current subtask type on different nodes. Average memory usage and average bandwidth usage , calculate the node matching degree of each subtask :

[0072]

[0073] sort the node matching degree of each subtask from large to small, output the node matching degree sorting sequence number of each subtask, and the smaller the sorting sequence number, the stronger the affinity of the subtask corresponding to the node.

[0074] Step 303, obtain the affinity processing subtask type sorting sequence number of each node and the node matching degree sorting sequence number of each subtask, multiply them to obtain a comprehensive affinity coefficient, output the node and subtask corresponding to the smallest comprehensive affinity coefficient for execution allocation, and then re-analyze the comprehensive affinity coefficients of the remaining subtasks and nodes for execution allocation until all subtasks are executed and allocated.

[0075] In use, the contents in steps 301 to 303 are combined:

[0076] obtain the real-time CPU utilization, real-time memory utilization and real-time bandwidth utilization of each node, combine the historical average CPU utilization of the current subtask type in different nodes , average memory usage and average bandwidth usage , calculate the node matching degree of each subtask , sort from large to small, output the node matching degree sorting sequence number of each subtask, multiply the affinity processing subtask type sorting sequence number of each node to obtain a comprehensive affinity coefficient, output the node and subtask corresponding to the smallest comprehensive affinity coefficient for execution allocation, and then re-analyze the comprehensive affinity coefficients of the remaining subtasks and nodes for execution allocation until all subtasks are executed and allocated.

[0077] Step four, obtain the real-time CPU remaining rate of each node , sum to obtain the real-time CPU remaining rate of the open distributed system, and if the real-time CPU remaining rate of the open distributed system is greater than 1, then take the integer part after rounding down as the number of rest nodes, and according to the number of rest nodes, close the corresponding number of nodes with the rear CPU performance ranking.

[0078] The step four includes the following contents:

[0079] Step 401, obtain the real-time CPU remaining rate of each node ​, and if the sum of the real-time CPU residual rates of the open distributed system is greater than 1, then the down-rounded value is recorded as the number of restable nodes.

[0080] Step 402, according to the number of restable nodes, the nodes with lower CPU performance ranking are closed.

[0081] The CPU performance ranking is realized by a CPU performance testing tool.

[0082] In use, the contents in steps 401 and 402 are combined.

[0083] The real-time CPU residual rates of the nodes are obtained. , and if the sum of the real-time CPU residual rates of the open distributed system is greater than 1, then the down-rounded value is recorded as the number of restable nodes, according to the number of restable nodes, the nodes with lower CPU performance ranking are closed, the closing of the nodes with lower performance ranking can reduce unnecessary power consumption, directly reduce operating costs, reduce the running time of low-load nodes, and reduce the hardware aging speed.

[0084] Please refer to Figure 2 The application provides an open distributed system resource management and task scheduling system, which comprises:

[0085] An initial data arrangement module obtains hardware resource data, software environment data and network condition data of each node, and obtains real-time state data of each node through a system monitoring tool, a user submits a task through a scheduling center, the global task is automatically decomposed into subtask units, and the subtask types are marked.

[0086] A node affinity analysis module obtains each subtask type, extracts the average processing time, average CPU utilization, average memory usage and average bandwidth usage of historical type subtasks in different nodes from node historical running logs, and calculates the running performance coefficient of different types of subtasks running on the same node After sorting from small to large, the affinity subtask type ranking sequence number of each node is output.

[0087] A task matching module obtains the real-time CPU utilization, real-time memory utilization and real-time bandwidth utilization of each node, and combines the historical average CPU utilization , average memory usage and average bandwidth usage of the current subtask type in different nodes to calculate the node matching degree of each subtask In descending order, output the node matching degree ranking number of each subtask, multiply the affinity processing subtask type ranking number of each node to obtain a comprehensive affinity coefficient, output the node and subtask corresponding to the minimum comprehensive affinity coefficient for execution allocation, and the execution allocation of all subtasks is completed.

[0088] The system redundancy analysis module obtains real-time CPU residual rates of each node The sum of the real-time CPU residual rates of the open distributed system is obtained, and if the sum of the real-time CPU residual rates of the open distributed system is greater than 1, then the number of nodes that can rest is recorded as an integer after being rounded down, and the number of nodes corresponding to the rear of the CPU performance ranking is closed according to the number of nodes that can rest.

[0089] The present application provides another embodiment:

[0090] The real-time CPU residual rates of each node are obtained, 0.2, 0.5, 0.6, 0.8, and 0.4, and the sum of the real-time CPU residual rates of the open distributed system is obtained, 0.2+0.5+0.6+0.8+0.4=2.5.

[0091] The sum of the real-time CPU residual rates of the open distributed system is greater than 1, and the number of nodes that can rest is recorded as 2 after being rounded down, and the two nodes corresponding to the rear of the CPU performance ranking are closed.

[0092] The above embodiments can be realized in whole or in part by software, hardware, firmware or any combination thereof. When realized by software, the above embodiments can be realized in whole or in part in the form of a computer program product. Those skilled in the art can realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution.

[0093] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Part or all of the units can be selected to achieve the purpose of the embodiment according to actual needs.

[0094] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any skilled person in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application.

Claims

1. An open distributed system resource management and task scheduling method, characterized in that: Comprising the following steps: Obtain each node hardware resource data, software environment data and network condition data, and obtain each node real-time state data through system monitoring tools, users submit tasks through the scheduling center, automatically decompose global tasks into subtask units, and mark the subtask type; Obtaining each sub-task type, extracting the average processing time, average CPU utilization, average memory usage and average bandwidth usage of the historical type sub-task in different nodes from the historical running log of the nodes, and calculating the running performance coefficient of different types of sub-tasks running on the same node After sorting from small to large, output the affinity sub-task type sorting sequence number of each node Obtaining real-time CPU utilization, real-time memory utilization and real-time bandwidth utilization of each node, combining historical average CPU utilization, average memory usage and average bandwidth usage of the current subtask type in different nodes to calculate node matching degree of each subtask , multiplying the node matching degree sorting sequence number of each subtask with the affinity subtask type sorting sequence number of each node to obtain a comprehensive affinity coefficient, outputting the node and subtask corresponding to the smallest comprehensive affinity coefficient for execution allocation, until all subtasks are executed and allocated.​​​ Obtaining real-time CPU residual rate of each node , summing up to obtain an open distributed system real-time CPU residual rate sum, if the open distributed system real-time CPU residual rate sum is greater than 1, then taking the integer part to record as a restable node number, and according to the restable node number, shutting down the node with the rear CPU performance ranking.

2. The open distributed system resource management and task scheduling method according to claim 1, characterized in that: Obtaining each subtask type, extracting the average processing time of the historical type subtask on different nodes from the historical running log of the node , average CPU utilization , average memory usage and average bandwidth usage , calculating the running performance coefficient of different types of subtasks running on the same node : Wherein, i represents the sequential number of each node of the open distributed system, i = 1, 2, …, n, n represents the total number of open distributed system nodes, j represents the sequential number of the subtask type, j = 1, 2, 3.

3. The open distributed system resource management and task scheduling method according to claim 1, characterized in that: The running performance coefficient of the same node running different types of sub-tasks Sort from small to large, output the affinity sub-task type ranking number of each node, the smaller the ranking number, the stronger the affinity of the type sub-task to the node.

4. The open distributed system resource management and task scheduling method according to claim 1, characterized in that: Obtaining real-time CPU utilization, real-time memory utilization and real-time bandwidth utilization of each node, and converting into real-time CPU remaining rate, real-time memory remaining amount and real-time bandwidth remaining amount ​​​ 。 5. The open distributed system resource management and task scheduling method according to claim 1, characterized in that: Based on the historical average CPU utilization of the current subtask type on different nodes Average memory usage and average bandwidth usage Calculate the node matching degree for each subtask. : 。 6. The open distributed system resource management and task scheduling method according to claim 1, characterized in that: Match the node affinity of each subtask Sort from large to small, output the node affinity sorting sequence number of each subtask, the smaller the sorting sequence number, the stronger the affinity of the corresponding subtask.

7. The open distributed system resource management and task scheduling method according to claim 1, characterized in that: Obtain the affinity subtask type ordering number of each node and the node matching degree ordering number of each subtask, multiply to obtain the comprehensive affinity coefficient, output the node and subtask corresponding to the smallest comprehensive affinity coefficient for execution allocation, and then reanalyze the comprehensive affinity coefficients of the remaining subtasks and nodes for execution allocation until all subtasks are executed and allocated.

8. An open distributed system resource management and task scheduling system for implementing the method of any one of claims 1 to 7, characterized by: Comprise: An initial data processing module, which obtains each node hardware resource data, software environment data and network condition data, and obtains each node real-time state data through system monitoring tools, users submit tasks through the scheduling center, automatically decompose global tasks into subtask units, and mark the subtask type; The node affinity analysis module obtains each subtask type, extracts the average processing time, the average CPU utilization, the average memory usage and the average bandwidth usage of the historical type subtasks on different nodes from the historical running log of the nodes, and calculates the running performance coefficient of different types of subtasks running on the same node After sorting from small to large, output the affinity subtask type sorting sequence number of each node; The task matching module obtains real-time CPU utilization, real-time memory utilization and real-time bandwidth utilization of each node, combines historical average CPU utilization, average memory usage and average bandwidth usage of the current subtask type in different nodes to calculate node matching degrees of each subtask The node matching degrees of each subtask are sorted from large to small, and the sorted sequence number of each subtask is multiplied by the sorted sequence number of the affinity subtask type of each node to obtain a comprehensive affinity coefficient. The node and the subtask corresponding to the smallest comprehensive affinity coefficient are executed and allocated, and the execution and allocation of all subtasks are completed.​​​ A system redundancy analysis module obtains real-time CPU remaining rates of each node , and obtains a sum of the real-time CPU remaining rates of the open distributed system, and if the sum of the real-time CPU remaining rates of the open distributed system is greater than 1, then the sum is rounded down to be recorded as a number of restable nodes, and according to the number of restable nodes, a number of nodes with a rear CPU performance ranking are closed.

Images