Distributed computing resource management system

Abstract

The invention discloses a distributed computing resource management system, and relates to the technical field of computing resource management, and the system comprises a client data collection module which is used for collecting task data submitted by a user at a client; the node query module is used for carrying out search processing on positions of nodes of data required by the task data according to the task data and the metadata to generate task metadata; according to the distributed computing resource management system, through a client data acquisition module, a node query module, a task data splitting module, a node resource acquisition module and a task waiting time computing module, a task can be split into a plurality of sub-tasks, the sub-tasks are allocated to corresponding nodes, and then the nodes compute and process the sub-tasks; the problems that the data transmission quantity is increased due to the fact that the nodes corresponding to the computing resources are different from the nodes stored in the task information, and the computing processing efficiency is reduced due to the fact that network fluctuation influences are likely to happen are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of computing resource management, and in particular to a distributed computing resource management system. Background Art

[0002] A distributed system is a system composed of multiple independent computers working together over a network. Each node uses message passing to share resources and collaborate on tasks. Its core goal is to improve performance, reliability, and scalability through parallel processing, while avoiding single points of failure. Distributed computing is an implementation of this system, breaking large-scale tasks into subtasks, assigning them to different nodes for execution, and then integrating the results. Typical applications include cloud computing, big data processing (such as Hadoop), and blockchain technology. The prior art with publication number CN117539613A discloses a method for managing shared resources in a central resource management center, which relates to the field of voltage control technology. Before the central resource management center allocates resources, it calculates the operating coefficient corresponding to each task node to obtain the priority level corresponding to each task node. When multiple task nodes request the same resource at the same time, the central resource management center allocates resources according to the priority level corresponding to each task node, and then in the order of the priority level corresponding to each task node. By analyzing the resource type matching coefficient corresponding to the central resource management center and each task node, and the resource matching coefficient between the central resource management center and each task node, it is beneficial for the central resource management center to allocate resources to each task node. By calculating the network fluctuation value corresponding to each task node, the problem of the central resource management center being unable to allocate resources due to network fluctuations during the resource allocation process is effectively solved.

[0003] However, the existing technology still has the problem of needing to uniformly allocate computing resources of different nodes according to the requests of the nodes. When the number of computing resource requests is large and the number of tasks processed in parallel by the system is large, it may happen that when a storage node requests computing resources, the node corresponding to the computing resources allocated is far away, resulting in a large delay in data transmission and greater susceptibility to network fluctuations, thereby reducing the efficiency of task processing calculations. Summary of the Invention

[0004] The purpose of the present invention is to provide a distributed computing resource management system to solve the above-mentioned deficiencies in the prior art.

[0005] In order to achieve the above object, the present invention provides the following technical solutions: a distributed computing resource management system, comprising a client data collection module for collecting task data submitted by users on the client; The node query module is used to search and process the location of the nodes required by the task data based on the task data and metadata, and generate the task metadata; The task data splitting module is used to split the task data based on the task metadata and generate subtask data; Node resource collection module, used to collect node resource configuration data and node resource occupancy data; The task waiting time calculation module is used to calculate the waiting time required for the node to complete the subtask data based on the subtask data, node resource configuration data and node resource occupancy data, and generate the subtask waiting time data; The node selection module is used to analyze and process the nodes that execute subtasks when the task waiting time is less than 1 hour based on the subtask data and the subtask waiting time data, generate minimum waiting time node analysis data, and perform node selection management operations based on the minimum waiting time node analysis data.

[0006] Furthermore, the client data collection module collects task data, including the following steps: S11. The task data A submitted by the user is collected through the client. Specifically, the user can upload the task code and configuration package to the system through the client, and then the task data A is parsed by the resource scheduler to obtain task dependencies and resource requirements.

[0007] Furthermore, the node query module generates task metadata, including the following steps: S21. Collect metadata through metadata services , , represents the metadata corresponding to the qth node, Indicates the maximum number of nodes. Metadata is used to record the logical location, permissions, version, and other meta-information of the data. That is, the node where the client requests the data can be found through metadata; S22: Based on task data A and metadata B, perform a search process for metadata B that matches task data A to generate a task metadata set. , , 、 Respectively represent the metadata corresponding to the q1th and q2th nodes, and the task metadata set There can be metadata of multiple identical nodes, indicating that a certain segment of data in task data A is stored in multiple nodes; further, consistent hashing or CRUSH algorithm can be used for searching.

[0008] Furthermore, the task data splitting module generates subtask data, including the following steps: S31. Based on task metadata collection Divide the task data A into to One-to-one correspondence between subtasks to generate subtask data sets , , Represents the w-th subtask data, Indicates the maximum number of subtask data, where Corresponding to at least one task metadata , that is, based on subtask data and its corresponding task metadata , you can build a subtask metadata collection , .

[0009] Furthermore, the node resource collection module collects node resource configuration data and node resource occupancy data, including the following steps: S41. Collect node resource configuration data and generate a node resource configuration data set , Indicates the resource configuration data of the qth node. The node resource configuration data includes CPU performance information, memory performance information, disk IO performance information, network bandwidth information, etc. S42. Collect node resource occupancy data and generate a node resource occupancy data set , Indicates the resource usage data of the qth node. Node resource usage data includes CPU usage, memory usage, I / O latency, IOPS / throughput, continuous bandwidth usage, node request queue length (i.e., the number of backlogged unprocessed requests), request latency, timeout / error rate, etc.

[0010] Furthermore, the task waiting time calculation module generates subtask waiting time data, including the following steps: S51. Based on subtask data , node resource configuration data and node resource usage data , perform node completion subtask data Calculate the waiting time required to generate a subtask waiting time data set , represents the w-th subtask waiting time data set, 、 Respectively represent the waiting time data of the subtask corresponding to the q3th and q4th nodes, where q3 can be equal to q4. When q3 is equal to q4, it means that there is only one corresponding node for the subtask; in one embodiment, the waiting time can be calculated by referring to the following method: Based on task information A, the computational requirements of the task are obtained, including computational effort (estimate the total number of operations (e.g., floating-point operations FLOPs) or data volume (e.g., GB) required for the task; task type (identify whether the task is compute-intensive, memory-intensive, or I / O-intensive); and algorithmic complexity (e.g., time complexity of O(n³) affects the time required as the data scale increases).

[0011] Configure data based on node resources and node resource usage data Get the node's configuration data, including CPU performance such as core count, clock frequency, memory bandwidth, storage speed, GPU acceleration, etc. For CPU-intensive tasks, you can refer to the following formula: ; For memory / bandwidth intensive tasks, you can refer to the following formula: ; Then, the time required for the task currently queued by the node is calculated using the formula, and the calculated time required for the current task is added to obtain the waiting time required for the node to process the current task.

[0012] Furthermore, the node selection module generates minimum waiting time node analysis data, including the following steps: S61, based on subtask waiting time data collection , minimize the sum of the waiting time for task data A and construct the objective function ; S62, based on subtask waiting time data collection , build a data set of total waiting time for task data , , Represents the pth combination of the total waiting time data of the task data, Indicates the maximum number of combinations of total waiting time data for task data, , Indicates the subtask waiting time data when the w-th subtask selects the q-th node; S63, search for the objective function in the task data total waiting time data set E search space Minimum task data total waiting time data , generate minimum waiting time node analysis data , including the following steps: S631, initialize algorithm parameters, number of ants N, maximum number of iterations T, algorithm parameters including pheromone volatility coefficient (usually 0< <1) Importance of pheromones , the importance of heuristic factors , pheromone enhancement constant ; S632. Each ant starts from the starting point and decides the direction of the next move based on the current location and unvisited points through the probability selection rule, gradually constructs the complete path, and calculates the distance of the complete path That is, calculate the total waiting time data of the task data corresponding to the complete path , the probability selection rule formula is as follows: , in, It represents the probability that ant k currently located at point i will choose the next point j. represents the pheromone concentration of path (i, j) at the tth iteration, is the heuristic factor, which is usually the inverse of the path (i, j) distance. Corresponding The reciprocal of , u is the point that ant k has not visited; S633. Simulate the natural evaporation of pheromone and update the pheromone concentration. The formula is as follows: ; S634. Simulate ants releasing pheromones on the path (i, j). The pheromone increment formula is as follows: If the path (i, j) is in the complete path of ant k On, the formula is: , Otherwise, the formula is: ; Get the full path The shortest ant releases pheromone, and the total pheromone update formula is as follows: ; S635: If the maximum number of iterations T is reached, the total waiting time data of the task data corresponding to the complete path at this time is output. , generate minimum waiting time node analysis data , otherwise return to step S632.

[0013] 1. Compared with the prior art, the distributed computing resource management system provided by the present invention can split a task into multiple subtasks and assign them to corresponding nodes through a client data acquisition module, a node query module, a task data splitting module, a node resource acquisition module, a task waiting time calculation module, and a node selection module, and then the nodes perform computing and processing on the subtasks, thereby avoiding the problem that the node corresponding to the computing resource is different from the node storing the task information, which increases the data transmission volume and is easily affected by network fluctuations, thereby reducing computing and processing efficiency.

[0014] 2. Compared with the existing technology, the distributed computing resource management system provided by the present invention calculates the waiting time required for each node to calculate the subtask, and then sums them up to obtain the total waiting time required for the task. It also searches for the node corresponding to the subtask when the total waiting time is the shortest by simulating ants releasing pheromones to select paths, and allocates subtasks according to the nodes corresponding to the searched subtasks, thereby ensuring the efficiency of task processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments described in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0016] Figure 1 This is a system structure block diagram provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0018] Example embodiments will be described more fully hereinafter with reference to the accompanying drawings, but the example embodiments may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the scope of this disclosure to those skilled in the art.

[0019] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0020] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0021] The terms used herein are used only to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a," "an," and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "comprising" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements, and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or groups thereof is not excluded.

[0022] The embodiments described herein may be described with reference to plan views and / or cross-sectional views, with the aid of idealized schematic diagrams of the present disclosure. Thus, the example illustrations may be modified based on manufacturing techniques and / or tolerances. Therefore, the embodiments are not limited to the embodiments shown in the accompanying drawings, but include modifications of the configurations formed based on the manufacturing process. Therefore, the regions illustrated in the accompanying drawings are schematic in nature, and the shapes of the regions shown in the drawings illustrate specific shapes of the regions of the elements, but are not intended to be limiting.

[0023] See also Figure 1 ,A distributed computing resource management system includes a client data acquisition module for ,acquiring task data submitted by a user at the client; The node query module is used to search and process the location of the nodes required by the task data based on the task data and metadata, and generate the task metadata; The task data splitting module is used to split the task data based on the task metadata and generate subtask data; Node resource collection module, used to collect node resource configuration data and node resource occupancy data; The task waiting time calculation module is used to calculate the waiting time required for the node to complete the subtask data based on the subtask data, node resource configuration data and node resource occupancy data, and generate the subtask waiting time data; The node selection module is used to analyze and process the nodes that execute subtasks when the task waiting time is less than 1 hour based on the subtask data and the subtask waiting time data, generate minimum waiting time node analysis data, and perform node selection management operations based on the minimum waiting time node analysis data.

[0024] The system provided by the present invention works by the following steps: The client data collection module collects task data, including the following steps: S11. The task data A submitted by the user is collected through the client. Specifically, the user can upload the task code and configuration package to the system through the client, and then the task data A is parsed by the resource scheduler to obtain task dependencies and resource requirements.

[0025] The node query module generates task metadata, including the following steps: S21. Collect metadata through metadata services , , represents the metadata corresponding to the qth node, Indicates the maximum number of nodes. Metadata is used to record the logical location, permissions, version, and other meta-information of the data. That is, the node where the client requests the data can be found through metadata; S22: Based on task data A and metadata B, perform a search process for metadata B that matches task data A to generate a task metadata set. , , 、 Respectively represent the metadata corresponding to the q1th and q2th nodes, and the task metadata set There can be metadata of multiple identical nodes, indicating that a certain segment of task data A is stored in multiple nodes; consistent hashing or CRUSH algorithm can be used for searching.

[0026] The task data splitting module generates subtask data, including the following steps: S31. Based on task metadata collection Divide the task data A into to One-to-one correspondence between subtasks to generate subtask data sets , , Represents the w-th subtask data, Indicates the maximum number of subtask data, where Corresponding to at least one task metadata , that is, based on subtask data and its corresponding task metadata , you can build a subtask metadata collection , .

[0027] The node resource collection module collects node resource configuration data and node resource occupancy data, including the following steps: S41. Collect node resource configuration data and generate a node resource configuration data set , Indicates the resource configuration data of the qth node. The node resource configuration data includes CPU performance information, memory performance information, disk IO performance information, network bandwidth information, etc. S42. Collect node resource occupancy data and generate a node resource occupancy data set , Indicates the resource usage data of the qth node. Node resource usage data includes CPU usage, memory usage, I / O latency, IOPS / throughput, continuous bandwidth usage, node request queue length (i.e., the number of backlogged unprocessed requests), request latency, timeout / error rate, etc.

[0028] In one example, a CPU utilization rate consistently exceeding 70-80% may indicate a node CPU overload; a memory usage rate consistently exceeding 80-90% or frequently triggering OOM (Out-of-Memory) events may indicate a node memory overload; an average disk I / O latency response time exceeding 10ms (HDD) or 1ms (SSD), or IOPS / throughput approaching the disk hardware limit, may indicate a node disk overload; and a node bandwidth usage consistently exceeding 70% may indicate a node network overload. All of these indicators indicate a node overload, resulting in a reduced ability to process tasks, increased processing time, or even unresponsiveness.

[0029] The task waiting time calculation module generates subtask waiting time data, including the following steps: S51. Based on subtask data , node resource configuration data and node resource usage data , perform node completion subtask data Calculate the waiting time required to generate a subtask waiting time data set , represents the w-th subtask waiting time data set, 、 Respectively represent the waiting time data of the subtask corresponding to the q3th and q4th nodes, where q3 can be equal to q4. When q3 is equal to q4, it means that there is only one corresponding node for the subtask; in one embodiment, the waiting time can be calculated by referring to the following method: Based on task information A, the computational requirements of the task are obtained, including computational effort (estimate the total number of operations (e.g., floating-point operations FLOPs) or data volume (e.g., GB) required for the task; task type (identify whether the task is compute-intensive, memory-intensive, or I / O-intensive); and algorithmic complexity (e.g., time complexity of O(n³) affects the time required as the data scale increases).

[0030] Configure data based on node resources and node resource usage data Get the node's configuration data, including CPU performance such as core count, clock frequency, memory bandwidth, storage speed, GPU acceleration, etc. For CPU-intensive tasks, you can refer to the following formula: ; For memory / bandwidth intensive tasks, you can refer to the following formula: ; Then, the time required for the task currently queued by the node is calculated using the formula, and the calculated time required for the current task is added to obtain the waiting time required for the node to process the current task.

[0031] The node selection module generates minimum waiting time node analysis data, including the following steps: S61, based on subtask waiting time data collection , minimize the sum of the waiting time for task data A and construct the objective function ; S62, based on subtask waiting time data collection , build a data set of total waiting time for task data , , Represents the pth combination of the total waiting time data of the task data, Indicates the maximum number of combinations of total waiting time data for task data, , Indicates the subtask waiting time data when the w-th subtask selects the q-th node; S63, search for the objective function in the task data total waiting time data set E search space Minimum task data total waiting time data , generate minimum waiting time node analysis data , including the following steps: S631, initialize algorithm parameters, number of ants N, maximum number of iterations T, algorithm parameters including pheromone volatility coefficient (usually 0< <1) Importance of pheromones , the importance of heuristic factors , pheromone enhancement constant ; S632. Each ant starts from the starting point and decides the direction of the next move based on the current location and unvisited points through the probability selection rule, gradually constructs the complete path, and calculates the distance of the complete path That is, calculate the total waiting time data of the task data corresponding to the complete path , the probability selection rule formula is as follows: , in, It represents the probability that ant k currently located at point i will choose the next point j. represents the pheromone concentration of path (i, j) at the tth iteration, is the heuristic factor, which is usually the inverse of the path (i, j) distance. Corresponding The reciprocal of , u is the point that ant k has not visited; S633. Simulate the natural evaporation of pheromone and update the pheromone concentration. The formula is as follows: ; S634. Simulate ants releasing pheromones on the path (i, j). The pheromone increment formula is as follows: If the path (i, j) is in the complete path of ant k On, the formula is: , Otherwise, the formula is: ; Get the full path The shortest ant releases pheromone, and the total pheromone update formula is as follows: ; S635: If the maximum number of iterations T is reached, the total waiting time data of the task data corresponding to the complete path at this time is output. , generate minimum waiting time node analysis data , otherwise return to step S632.

[0032] S64, then analyze the data based on the minimum waiting time node The node corresponding to each subtask in the , assigns the subtask to the corresponding node, and completes the node selection management task.

[0033] The above description is merely illustrative of certain exemplary embodiments of the present invention. It goes without saying that those skilled in the art will be able to modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and description are illustrative in nature and should not be construed as limiting the scope of protection of the claims.

Claims

1. A distributed computing resource management system, characterized by: It includes a client data collection module for collecting task data submitted by users on the client; A node query module is used to search and process the location of the node required by the task data based on the task data and metadata, and generate task metadata; A task data splitting module is used to split the task data based on the task metadata to generate subtask data; Node resource collection module, used to collect node resource configuration data and node resource occupancy data; A task waiting time calculation module is used to calculate the waiting time required for a node to complete the subtask data based on the subtask data, node resource configuration data and node resource occupancy data, and generate subtask waiting time data; The node selection module is used to analyze and process the nodes that execute subtasks when the task waiting time is less than 1 hour based on the subtask data and the subtask waiting time data, generate minimum waiting time node analysis data, and perform node selection management operations based on the minimum waiting time node analysis data.

2. A distributed computing resource management system according to claim 1, characterized in that: The client data collection module collects the task data, including the following steps: S11. Collect task data A submitted by the user through the client.

3. A distributed computing resource management system according to claim 2, characterized in that: The node query module generates task metadata, including the following steps: S21. Collect metadata through metadata services , , represents the metadata corresponding to the qth node, Indicates the maximum number of nodes; S22: Based on the task data A and the metadata B, perform a search process for the metadata B that matches the task data A to generate a task metadata set. , , 、 Represents the metadata corresponding to the q1th and q2th nodes respectively.

4. A distributed computing resource management system according to claim 3, characterized in that: The task data splitting module generates subtask data, including the following steps: S31, based on the task metadata set Divide the task data A to generate subtask data sets , , Represents the w-th subtask data, Indicates the maximum number of subtask data, where Corresponding to at least one task metadata .

5. A distributed computing resource management system according to claim 4, characterized in that: The node resource collection module collects node resource configuration data and node resource occupancy data, including the following steps: S41. Collect node resource configuration data and generate a node resource configuration data set , Represents the resource configuration data of the qth node; S42. Collect node resource occupancy data and generate a node resource occupancy data set , Indicates the resource usage data of the qth node.

6. A distributed computing resource management system according to claim 5, characterized in that: The task waiting time calculation module generates subtask waiting time data, including the following steps: S51. Based on subtask data , node resource configuration data and node resource usage data , the node completes the subtask data Calculate the waiting time required to generate a subtask waiting time data set .

7. A distributed computing resource management system according to claim 6, characterized in that: The node selection module generates minimum waiting time node analysis data, including the following steps: S61, based on subtask waiting time data collection , minimize the sum of the waiting time for task data A and construct the objective function ; S62, based on subtask waiting time data collection , build a data set of total waiting time for task data , , Represents the pth combination of the total waiting time data of the task data, Indicates the maximum number of combinations of total waiting time data for task data, , Indicates the subtask waiting time data when the w-th subtask selects the q-th node; S63, searching for the task data total waiting time data set E search space so that the objective function Minimum task data total waiting time data , generate minimum waiting time node analysis data , including the following steps: S631, initialize algorithm parameters, number of ants N, maximum number of iterations T, the algorithm parameters include pheromone volatility coefficient (usually 0< <1) Importance of pheromones , the importance of heuristic factors , pheromone enhancement constant ; S632. Each ant starts from the starting point and decides the direction of the next move based on the current location and unvisited points through the probability selection rule, gradually constructs the complete path, and calculates the distance of the complete path That is, calculate the total waiting time data of the task data corresponding to the complete path , the probability selection rule formula is as follows: , in, It represents the probability that ant k currently located at point i will choose the next point j. represents the pheromone concentration of path (i, j) at the tth iteration, is the heuristic factor, u is the point not visited by ant k; S633. Simulate the natural evaporation of pheromone and update the pheromone concentration. The formula is as follows: ; S634. Simulate ants releasing pheromones on the path (i, j). The pheromone increment formula is as follows: If the path (i, j) is in the complete path of ant k On, the formula is: , Otherwise, the formula is: ; Let the full path The shortest ant releases pheromone, and the total pheromone update formula is as follows: ; S635: If the maximum number of iterations T is reached, the total waiting time data of the task data corresponding to the complete path at this time is output. , generate minimum waiting time node analysis data , otherwise return to step S632.

Images