Self-adaptive resource scheduling method for improving cluster reconstruction efficiency

Abstract

The invention discloses a self-adaptive resource scheduling method for improving cluster reconstruction efficiency, and relates to the field of resource scheduling distribution, and the method comprises the steps: monitoring the resource use data of each node in a cluster in real time through a monitoring agent, and obtaining the health state of the node through a heartbeat mechanism; collecting input task requests, predicting the resource demand quantity of each task, and performing preliminary queue processing on the tasks to obtain a task queue group; setting a resource competition constraint according to the task queue group and the resource use data and the health state of each node, obtaining dynamic resource scheduling corresponding to each task by using an optimization algorithm, and allocating the tasks to each node; a resource scheduling dynamic graph is constructed, a visual interface is set, and the resource scheduling dynamic graph is arranged on the visual interface and used for displaying resource scheduling, providing a user-friendly interface and realizing high-efficiency scheduling of adaptive resources in a cluster reconstruction process.

Description

Technical Field

[0001] The present invention relates to the field of resource scheduling and allocation, and more specifically, to an adaptive resource scheduling method for improving cluster reconstruction efficiency. Background Art

[0002] Cluster reconstruction refers to the process of rebuilding, repairing or restoring an existing cluster in a computer system. A cluster usually consists of multiple nodes (computers) that work together over a network to provide high availability, high performance or scalability services. When some nodes in the cluster fail, data is corrupted, hardware or software needs to be upgraded, or the configuration needs to be changed, the cluster may need to be rebuilt.

[0003] In the process of improving cluster reconstruction efficiency, adaptive resource scheduling will encounter problems such as task scheduling delays and increased scheduling overhead. For failed nodes, the scheduler needs to quickly detect the failure and reallocate tasks, especially in scenarios where the cluster is large or the network latency between nodes is high. Multiple tasks request the same resources (such as CPU, memory, bandwidth, etc.) at the same time, which may lead to resource contention, resulting in performance degradation, and even task failure or retries, extending the cluster reconstruction time. In addition, the failure of nodes is uncontrollable and random, so node fault detection needs to be performed at all times.

[0004] Therefore, based on the above problems, it is necessary to design adaptive resource scheduling to improve cluster reconstruction efficiency. Summary of the invention

[0005] In view of the problems existing in the prior art, the object of the present invention is to provide an adaptive resource scheduling method for improving cluster reconstruction efficiency, so as to achieve efficient scheduling of adaptive resources during cluster reconstruction.

[0006] To achieve the above object, the present invention provides the following technical solution: The adaptive resource scheduling method for improving cluster reconstruction efficiency includes:

[0007] Step S1: Use the monitoring agent to monitor the resource usage data of each node in the cluster in real time, and obtain the health status of the node through the heartbeat mechanism;

[0008] Step S2: Collect input task requests, predict the resource requirements of each task, perform preliminary queue processing on the tasks, and obtain a task queue group;

[0009] Step S3: Set resource competition constraints according to the resource usage data and health status of the task queue group and each node, use the optimization algorithm to obtain the dynamic resource scheduling corresponding to each task, and assign the task to each node;

[0010] Step S4: construct a resource scheduling dynamic graph, set up a visualization interface, and arrange the resource scheduling dynamic graph on the visualization interface to display resource scheduling.

[0011] Preferably, the resource usage data includes CPU usage, memory usage, disk I / O and network bandwidth usage;

[0012] Health status includes normal state and abnormal state;

[0013] Deploy lightweight monitoring agents in parallel on each node, and the monitoring agents collect node resource usage in real time;

[0014] Within the preset time t, the sampling frequency fs is set, and then the sampling length Cy=t×fs is calculated; the collected resource usage data is normalized and standardized;

[0015] The collected resource usage data and health status are transmitted to the central monitoring server via the network for storage.

[0016] Preferably, the method for obtaining the health status of a node through a heartbeat mechanism includes:

[0017] Step Q1: Get the heartbeat interval of each node. For each node, when each heartbeat interval arrives, the monitoring agent sends a heartbeat request to the monitored node. The monitored node receives the heartbeat request and performs a corresponding action. After self-checking according to the corresponding action, it generates a heartbeat response and transmits the heartbeat response back to the monitoring agent.

[0018] Among them, the heartbeat request includes network status check, service status check, application running status check and database connection status check, and the heartbeat response includes health status and error information;

[0019] Step Q12: After receiving the heartbeat response, the monitoring agent analyzes it. If the heartbeat response indicates that the monitored node is in a normal state, the monitoring agent records and updates the health state, and sends the heartbeat request again after the heartbeat interval. If the heartbeat response indicates that the monitored node is in an abnormal state, the monitoring agent marks the node as abnormal and sends an alarm.

[0020] Step Q3: If the monitoring agent does not receive a heartbeat response from the monitored node within the specified time, it will resend the heartbeat request until it receives a heartbeat response or reaches the limit condition. If it receives a heartbeat response, it will be processed according to step Q2. If it reaches the limit condition, it will mark the node as disconnected and send an alarm. The limit condition is that the heartbeat request resent within the heartbeat interval does not exceed re times;

[0021] Step Q4: Collect the health status of all nodes and display them in summary.

[0022] Preferably, the method of obtaining the heartbeat interval of each node includes:

[0023] For each node, set the initial interval to Xt 0 , represents the initial interval of the qth node, and sets the update of the heartbeat interval to

[0024] in, It represents the iteratively updated heartbeat interval of the qth node when it sends the heartbeat request for the rth time. It represents the heartbeat interval that the qth node iteratively updates when it sends a heartbeat request for the r-1th time. It represents the speed of iterative update of the qth node at the rth time, represents the inertia weight of the qth node at the rth iteration update, ∈(r) represents the random adjustment item at the rth iteration update;

[0025] speed Obtained through resource usage data and historical health status of nodes,

[0026] in, represents the speed of iterative update of the qth node at the r-1th time, α1, α2, α3, α4 and α5 represent weight coefficients, S_cpu r 、S_mem r 、S_dis r 、S_net r and G_Z r Indicates the CPU usage, memory usage, disk I / O, network bandwidth usage, and number of node failures of the node during the rth iteration update, S_cpu r-1 、S_mem r-1 、S_dis r-1 、S_net r-1 and G_Z r-1 Indicates the CPU usage, memory usage, disk I / O, network bandwidth usage, and number of node failures of the node during the r-1th iteration update. and Indicates the node's standard CPU usage, memory usage, disk I / O, network bandwidth usage, and number of node failures;

[0027] Inertia Weight Calculate based on the changing trend of heartbeat intervals, Indicates the heartbeat interval that is iteratively updated when the qth node sends a heartbeat request for the r-2th time.

[0028] Preferably, the method for queue processing of tasks includes:

[0029] Extract historical task request data and use machine learning models to predict the resource requirements of each task; set initial priority levels based on the confidentiality, urgency, and resource requirements of each task Use the decay mechanism to update the priority of each task in, represents the initial priority level of the i-th task, Indicates the priority level of the i-th task at the j-th time, represents the priority level of the i-th task at the j-1th time, γ1, γ2 and γ3 represent the proportion coefficients, and Bm i Indicates the confidentiality level of the i-th task, Jm i Indicates the urgency of the i-th task, Zm i represents the resource requirement of the ith task, represents the growth coefficient, T i Indicates the waiting time of the i-th task not being processed;

[0030] Calculate the priority of each task in turn and arrange them in order from high to low.

[0031] Preferably, the method of using a machine learning model to predict the resource requirements of each task includes:

[0032] Collect historical task feature data, including task type, task size, task complexity, and the resource requirements of the corresponding tasks to form a sample set;

[0033] Normalize and standardize the data of the sample set before preprocessing;

[0034] Use the sample set to train the machine learning model, which is a neural network or deep learning model. Use the task feature data as the input of the model, and the resource demand as the output of the model. Set the mean square error as the loss function. When the loss function value is the minimum or no longer changes, the training stops and the trained machine learning model is obtained.

[0035] Input the task feature data of each current task into the trained machine learning model to obtain the resource requirements of each task.

[0036] Preferably, the method for setting resource competition constraints includes:

[0037] The resource competition constraints are set to include a first constraint, a second constraint and a third constraint, and the priority level is that the first constraint is greater than the second constraint, and the second constraint is greater than the third constraint;

[0038] The first constraint is that the nodes to which tasks are assigned are all in normal state, and are recorded as assignable nodes;

[0039] The second constraint is that the resource usage data of the allocatable nodes are all below the upper threshold;

[0040] The third constraint is that tasks of the same priority are allocated in descending order of resource requirements.

[0041] Preferably, the method of using an optimization algorithm to obtain dynamic resource scheduling corresponding to each task includes:

[0042] Under the condition of satisfying resource competition constraints, an optimization algorithm is used to allocate nodes;

[0043] Step D1: The optimization algorithm is based on the improved particle swarm algorithm. The number of nodes that can be assigned tasks is obtained. The same number of particle swarms is set according to the number of nodes, which is recorded as behavior particle swarms. Each task generates a behavior particle swarm.

[0044] Step D2: Set the starting position of the particle to the task queue group and the end position to the node where the task is stored. Use the fitness function to evaluate the quality of the path. The fitness function Among them, H_zy represents the remaining resource usage data of the node, R_zy represents the resource demand of the task, Kx represents the path cost, Yc represents the delay time, μ1 and μ2 represent the weight coefficients;

[0045] Step D3: With the goal of minimizing the fitness function, the tasks of the same priority level are calculated in turn according to the task queue group. For each task, the node with the smallest fitness function is selected as the allocation node, and the task is allocated to the corresponding node;

[0046] Step D4: Update the task queue group, repeat steps D1 to D3, and sequentially allocate nodes to the continuously updated task requests.

[0047] Preferably, the method for constructing the resource scheduling dynamic graph includes:

[0048] Resource scheduling dynamic graphs include task graphs and task allocation graphs;

[0049] The task chart includes establishing a blank two-dimensional coordinate chart, taking the time point as the horizontal coordinate and the task priority level as the vertical coordinate, and filling the task request at each time point into the corresponding position according to the task priority level;

[0050] The task allocation chart includes establishing a blank plane, simulating all task allocation nodes into first solid points, simulating the location where the task request occurs into second solid points, representing the first solid points and the second solid points with different colors, and filling them into the blank plane according to the actual positions, obtaining the nodes corresponding to each task, that is, the first solid points corresponding to each second solid point, connecting them through directed edges to display the corresponding allocation of tasks and nodes.

[0051] Preferably, the visualization interface is used to interact with the user and provide the user with multi-level visualization of the data.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] The adaptive resource scheduling method combines the resource demand prediction of the task and the resource usage of the node, and can dynamically adjust the resource allocation of the task, avoiding the overload or idleness of node resources, and greatly improving the utilization of resources. By introducing a machine learning model to predict the resource demand of the task, and setting the priority based on the urgency and confidentiality of the task, the resource demand of the task can be accurately determined, ensuring that the task is efficiently scheduled according to the priority.

[0054] The dynamic adjustment mechanism of task priority (such as the decay mechanism based on waiting time) ensures that high-priority tasks are processed first and that low-priority tasks can be scheduled in a timely manner over time, thereby improving overall scheduling efficiency.

[0055] The heartbeat mechanism monitors the health status of nodes in real time, detects and marks abnormal or disconnected nodes in a timely manner, prevents tasks from being assigned to abnormal nodes, and improves system reliability. The dynamic adjustment mechanism of the node heartbeat interval combines resource usage and historical health status, avoids excessive monitoring or delayed detection caused by fixed heartbeat intervals, and further improves the accuracy and response speed of node health status monitoring.

[0056] The improved particle swarm optimization algorithm (PSO) is used to allocate tasks to nodes. By minimizing the fitness function (taking into account factors such as resource surplus, task requirements, path overhead and delay time), the task is allocated to the most suitable node, optimizing the resource scheduling process. The setting of resource competition constraints (such as node normal status, resource usage upper limit, etc.) further ensures the rationality and fairness of resource scheduling and avoids resource conflicts and unreasonable allocation.

[0057] The constructed resource scheduling dynamic graph and visualization interface provide an intuitive task scheduling display, allowing users to clearly see the task execution status, resource allocation, and node health status, enhancing the system's operability and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A structural schematic diagram of an adaptive resource scheduling system for improving cluster reconstruction efficiency is proposed in the present invention;

[0059] Figure 2 It is a schematic diagram of a method of an adaptive resource scheduling system used in the present invention to improve cluster reconstruction efficiency. DETAILED DESCRIPTION

[0060] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0061] Embodiment 1

[0062] Reference Figure 1 and Figure 2 Embodiment 1 further illustrates an adaptive resource scheduling system for improving cluster reconstruction efficiency proposed by the present invention.

[0063] With the development of real-life 3D, the corresponding modeling software is installed in the cluster for parallel high-performance modeling. For example, when large-scale city-level fine modeling is performed, the CC project will have the problem of multi-node performance degradation when reconstructed on the cluster. There will be bottlenecks in the number of nodes in the CC project or storage. In particular, when there are too many task requests during reconstruction or multiple task requests appear at the same time, the reconstruction efficiency will stagnate or decline.

[0064] In response to the above technical problems, it is necessary to design a solution that can share the reading and writing pressure of multiple nodes on the same project, balance the performance bottleneck of the project under multi-node and multi-task requests, adjust the priority of tasks under the project, and realize regional priority production sorting. It can disperse the pressure of reading and writing image files to multiple projects and nodes, and at the same time balance and ensure that each node is utilized as much as possible.

[0065] In order to improve cluster reconstruction efficiency and solve problems such as resource contention and node failure, an adaptive resource scheduling method and system can be designed. The system should have intelligent resource allocation strategies, fault detection and recovery mechanisms, and dynamic task scheduling optimization capabilities. The following is a complete adaptive resource scheduling method and system design:

[0066] When dealing with node failures, the scheduler may not have sufficient flexibility and adaptability to make the best redundant scheduling decision based on the current resource status and task progress. Therefore, node failures are inevitable during cluster reconstruction. The scheduler needs to quickly detect failures and reallocate tasks. However, excessive redundant scheduling may increase resource overhead, while insufficient redundancy may lead to inefficient reconstruction.

[0067] Step S1: Use the monitoring agent to monitor the resource usage data of each node in the cluster in real time, and obtain the health status of the node through the heartbeat mechanism;

[0068] Resource usage data includes CPU usage, memory usage, disk I / O, and network bandwidth usage;

[0069] CPU usage is used to describe the CPU usage of each node. It can be measured by the occupancy of each core, that is, the ratio of used CPU time to total CPU time. "Used CPU time" refers to the total time spent by the CPU executing tasks during the monitoring period, and "total CPU time" refers to the total length of the monitoring period. The higher the CPU usage value, the busier the CPU is, and the hardware may need to be optimized or upgraded.

[0070] Memory usage is used to describe the memory usage of a node, including the proportion of physical memory and virtual memory, that is, the percentage of used memory to total available memory. "Used memory" refers to the amount of memory currently occupied by the operating system and applications, and "total available memory" refers to the total amount of memory installed on the node. The higher the memory usage value, the tighter the memory resources are, and you may need to increase the memory or optimize the memory usage.

[0071] Disk I / O is used to describe the read and write rate of the disk, reflecting the storage pressure of the node, that is, the ratio of the number of read and write bytes to the time interval, where "number of read and write bytes" refers to the amount of data read or written from the disk in a specific time interval, and "time interval" refers to the time range of measurement. The higher the disk I / O value, the more frequent the disk read and write operations are, and you may need to optimize the storage system or upgrade the hardware.

[0072] The network bandwidth utilization rate is used for the network inbound and outbound traffic of the mode node, including the utilization rate of upload and download bandwidth, that is, the ratio of the amount of transmitted data to the total bandwidth, where "transmitted data volume" refers to the amount of data transmitted through the network interface within a specific time interval, and "total bandwidth" refers to the maximum transmission capacity of the network interface. The higher the value of the network bandwidth utilization rate, the more constrained the network resources are, and it may be necessary to upgrade the network equipment or optimize the network traffic.

[0073] Health status includes normal state and abnormal state;

[0074] Lightweight monitoring agents are deployed in parallel on each node to collect node resource usage in real time, so as to fully cover voltage signals.

[0075] The monitoring agent is a lightweight software component deployed on each node, such as PrometheusNode Exporter, Collectd, and Telegraf. It is responsible for collecting monitoring data such as the resource usage and health status of the node in real time, and sending this data to the central monitoring server (hard disk or cloud disk storage).

[0076] Within a preset time t (eg, 1, 10 or 30 seconds), the sampling frequency fs (eg, 10 kHz) is set, and then the sampling length Cy=t×fs is calculated; (eg, Cy=1×10000=10000 points).

[0077] Normalize and standardize the collected resource usage data;

[0078] The collected resource usage data and health status are transmitted to the central monitoring server via the network for storage.

[0079] In addition to performance data, the monitoring agent can also collect information such as system logs and application logs on the node. Log information can help analyze abnormal conditions of the system or application.

[0080] Methods for obtaining the health status of a node through the heartbeat mechanism include:

[0081] Step Q1: Get the heartbeat interval of each node. The heartbeat interval is a fixed time interval for heartbeat requests sent regularly. For example, the heartbeat interval may be set to every 10 seconds or 30 seconds. For each node, when each heartbeat interval arrives, the monitoring agent sends a heartbeat request to the monitored node. The heartbeat request is usually a lightweight network message, which can be a simple HTTP request, TCP / UDP packet or other protocol request. This request usually contains the identification information of the monitoring agent and may contain the requesting node information so that the node can identify and respond. The monitored node receives the heartbeat request and performs the corresponding action. After self-checking according to the corresponding action, it generates a heartbeat response and transmits the heartbeat response back to the monitoring agent;

[0082] Among them, the heartbeat request includes network status check, service status check, application running status check and database connection status check. The network status check is used to check whether the network connection is normal, the service status check is used to check whether the key services are running normally, and the application running status check and database connection status check are similar to the above. The heartbeat response includes health status and error information; the health status includes normal status and abnormal status, and the error information is the error code or error log contained when the node finds an abnormality.

[0083] Step Q12: After receiving the heartbeat response, the monitoring agent analyzes it. If the heartbeat response indicates that the monitored node is in a normal state, it records and updates the health status, and sends a heartbeat request again after the heartbeat interval. If the heartbeat response indicates that the monitored node is in an abnormal state, it marks the node as abnormal and sends an alarm; the problem is recorded in the log or event system. For example, the administrator is notified or the automatic recovery mechanism is triggered by email, SMS, Webhooks, etc.

[0084] Step Q3: If the monitoring agent does not receive a heartbeat response from the monitored node within the specified time, it will resend the heartbeat request until it receives a heartbeat response or reaches the limit condition. If it receives a heartbeat response, it will be processed according to step Q2. If it reaches the limit condition, it will mark the node as disconnected and send an alarm. The limit condition is that the heartbeat request resent within the heartbeat interval does not exceed re times;

[0085] Step Q4: Collect the health status of all nodes and display them in summary.

[0086] Through the heartbeat mechanism, the monitoring system can continuously obtain the health status of the node and take timely measures when problems occur. The entire process from initialization to sending requests, node processing, response reception and analysis forms a complete closed loop to ensure high availability and stable operation of the system.

[0087] The heartbeat interval is very important for monitoring the health status of multiple nodes, especially when the operating status and load of each node are different. The goal of optimizing the heartbeat interval and frequency is to reduce resource consumption and avoid unnecessary monitoring overhead by dynamically adjusting the frequency of monitoring heartbeats, while ensuring that the health status of nodes can be checked more frequently when the system load is high or anomalies occur. The core idea of ​​the optimization is to intelligently adjust the frequency of heartbeats according to the system load and health status.

[0088] Get the heartbeat interval of each node. The specific methods include:

[0089] For each node, set the initial interval to Xt 0 , represents the initial interval of the qth node, and sets the update of the heartbeat interval to

[0090] in, It represents the iteratively updated heartbeat interval of the qth node when it sends the heartbeat request for the rth time. It represents the heartbeat interval that the qth node iteratively updates when it sends a heartbeat request for the r-1th time. It represents the speed of iterative update of the qth node at the rth time, represents the inertia weight of the qth node at the rth iteration update, ∈(r) represents the random adjustment item at the rth iteration update;

[0091] speed Obtained through resource usage data and historical health status of nodes,

[0092] in, represents the speed of iterative update of the qth node at the r-1th time, α1, α2, α3, α4 and α5 represent weight coefficients, S_cpu r 、S_mem r 、S_dis r 、S_net r and G_Z r Indicates the CPU usage, memory usage, disk I / O, network bandwidth usage, and number of node failures of the node during the rth iteration update, S_cpu r-1 、S_mem r-1 、S_dis r-1 、S_net r-1 and G_Z r-1 Indicates the CPU usage, memory usage, disk I / O, network bandwidth usage, and number of node failures of the node during the r-1th iteration update. and Indicates the node's standard CPU usage, memory usage, disk I / O, network bandwidth usage, and number of node failures;

[0093] and The setting is made based on experimental data analysis or experience.

[0094] Inertia Weight Calculate based on the changing trend of heartbeat intervals, Indicates the heartbeat interval that is iteratively updated when the qth node sends a heartbeat request for the r-2th time.

[0095] The heartbeat interval is dynamically adjusted according to the importance, load or network status of the node, and the heartbeat frequency is automatically adjusted to detect the health status with the maximum resource value, thereby reducing unnecessary monitoring frequency and saving resources.

[0096] In scenarios where the cluster is large or the network latency between nodes is high, a reasonable task queue sorting and resource scheduling mechanism is particularly important to deal with resource competition and scramble when multiple tasks request the same resources (such as CPU, memory, and bandwidth).

[0097] Step S2: Collect input task requests, predict the resource requirements of each task, perform preliminary queue processing on the tasks, and obtain a task queue group;

[0098] Methods for queueing tasks include:

[0099] Extract historical task request data and use machine learning models to predict the resource requirements of each task;

[0100] Set initial priorities for each mission based on confidentiality, urgency, and resource requirements Ensure that low-priority tasks will not be starved for a long time (starvation), and use the decay mechanism to update the priority level of each task That is, as the waiting time of the task increases, its priority is gradually increased, among which, represents the initial priority level of the i-th task, Indicates the priority level of the i-th task at the j-th time, represents the priority level of the i-th task at the j-1th time, γ1, γ2 and γ3 represent the proportion coefficients, and Bm i Indicates the confidentiality level of the i-th task, Jm i Indicates the urgency of the i-th task, Zm i represents the resource requirement of the ith task, represents the growth coefficient, T i Indicates the waiting time of the i-th task not being processed;

[0101] The confidentiality level and urgency level can be expressed by numerical values. For example, the confidentiality level is set to 1, 2, 3, 4 and 5. The larger the value, the higher the confidentiality requirement. The urgency level can also be set in the same way. For tasks with order requirements, the urgency level of the tasks in the earlier order is higher.

[0102] Calculate the priority of each task in turn and arrange them in order from high to low.

[0103] When the scheduler receives a new task, it first estimates the resources required for the task based on the prediction model to avoid over- or under-allocation of resources.

[0104] Use machine learning models to predict the resource requirements of each task. Specific methods include:

[0105] Collect historical task feature data, including task type, task size, task complexity, and the resource requirements of the corresponding tasks to form a sample set (such as CPU, memory, disk space, etc.). The data source can be log files, databases, monitoring systems, etc.

[0106] Normalize and standardize the data of the sample set before preprocessing;

[0107] Data normalization: Normalize the data to the same range to prevent the values ​​of some features from being too large or too small;

[0108] Use the sample set to train the machine learning model, which is a neural network or deep learning model. Use the task feature data as the input of the model, and the resource demand as the output of the model. Set the mean square error as the loss function. When the loss function value is the minimum or no longer changes, the training stops and the trained machine learning model is obtained.

[0109] Input the task feature data of each current task into the trained machine learning model to obtain the resource requirements of each task.

[0110] Step S3: Set resource competition constraints according to the resource usage data and health status of the task queue group and each node, use the optimization algorithm to obtain the dynamic resource scheduling corresponding to each task, and assign the task to each node;

[0111] The methods for setting resource competition constraints include:

[0112] The resource competition constraints are set to include a first constraint, a second constraint and a third constraint, and the priority level is that the first constraint is greater than the second constraint, and the second constraint is greater than the third constraint;

[0113] The first constraint is that the nodes to which tasks are assigned are all in normal state, and are recorded as assignable nodes;

[0114] The second constraint is that the resource usage data of the allocatable nodes are all below the upper threshold; for example, CPU usage is less than 60%, memory usage is less than 80%, disk I / O is less than 75%, network bandwidth usage is less than 75%, etc.

[0115] The third constraint is that tasks of the same priority are allocated in descending order of resource requirements.

[0116] Use the optimization algorithm to obtain the dynamic resource scheduling corresponding to each task. The specific methods include:

[0117] Under the condition of satisfying resource competition constraints, an optimization algorithm is used to allocate nodes;

[0118] Step D1: The optimization algorithm is based on the improved particle swarm algorithm. The number of nodes that can be assigned tasks is obtained. The same number of particle swarms is set according to the number of nodes, which is recorded as behavior particle swarms. Each task generates a behavior particle swarm.

[0119] Step D2: Set the starting position of the particle to the task queue group and the end position to the node where the task is stored. Use the fitness function to evaluate the quality of the path. The fitness function Among them, H_zy represents the remaining resource usage data of the node, R_zy represents the resource demand of the task, Kx represents the path cost, Yc represents the delay time, μ1 and μ2 represent the weight coefficients;

[0120] Step D3: With the goal of minimizing the fitness function, the tasks of the same priority level are calculated in turn according to the task queue group. For each task, the node with the smallest fitness function is selected as the allocation node, and the task is allocated to the corresponding node;

[0121] Step D4: Update the task queue group, repeat steps D1 to D3, and sequentially allocate nodes to the continuously updated task requests.

[0122] Step S4: construct a resource scheduling dynamic graph, set a visualization interface, and arrange the resource scheduling dynamic graph on the visualization interface to display resource scheduling;

[0123] The method for constructing a resource scheduling dynamic graph includes:

[0124] Resource scheduling dynamic graphs include task graphs and task allocation graphs;

[0125] The task chart includes establishing a blank two-dimensional coordinate chart, taking the time point as the horizontal coordinate and the task priority level as the vertical coordinate, and filling the task request at each time point into the corresponding position according to the task priority level;

[0126] The task allocation chart includes establishing a blank plane, simulating all task allocation nodes into first solid points, simulating the location where the task request occurs into second solid points, representing the first solid points and the second solid points with different colors, and filling them into the blank plane according to the actual positions, obtaining the nodes corresponding to each task, that is, the first solid points corresponding to each second solid point, connecting them through directed edges to display the corresponding allocation of tasks and nodes.

[0127] The visual interface is displayed through display hardware and provides interactive operations.

[0128] The present invention is suitable for task scheduling of large-scale clusters, can handle complex and diverse task requests, supports parallel scheduling and resource allocation of multiple tasks, and adapts to the needs of large-scale, high-concurrency environments.

[0129] The visual interactive interface also supports users to view and analyze scheduling data at multiple levels, making it easier for users to adjust and optimize according to actual needs.

[0130] Through the adaptive scheduling mechanism, the task allocation strategy can be dynamically adjusted according to the real-time request of the task and the resource usage of the node, which enhances the flexibility of task scheduling. When the task resource requirements and node resource status are constantly changing, the system can adaptively adjust resource allocation to ensure the real-time and responsiveness of scheduling.

[0131] Through technical means such as predicting task resource requirements through machine learning models, automatically updating task priorities, and dynamically adjusting node heartbeat intervals, the reliance on manual intervention is reduced, greatly improving the system's degree of automation and intelligence.

[0132] The technical solution of the present invention effectively improves the cluster reconstruction efficiency through technical means such as real-time monitoring, machine learning prediction, resource competition constraints and optimization algorithms, and achieves significant optimization effects in resource utilization, scheduling efficiency, system reliability, stability, etc., and is suitable for large-scale, complex and diverse cluster scheduling scenarios.

[0133] Embodiment 2

[0134] Reference Figure 1 and Figure 2 , Example 2 further illustrates an adaptive resource scheduling system for improving cluster reconstruction efficiency proposed by the present invention.

[0135] The adaptive resource scheduling method for improving cluster reconstruction efficiency comprises the following steps:

[0136] Step S1: Use the monitoring agent to monitor the resource usage data of each node in the cluster in real time, and obtain the health status of the node through the heartbeat mechanism;

[0137] Step S2: Collect input task requests, predict the resource requirements of each task, perform preliminary queue processing on the tasks, and obtain a task queue group;

[0138] Step S3: Set resource competition constraints according to the resource usage data and health status of the task queue group and each node, use the optimization algorithm to obtain the dynamic resource scheduling corresponding to each task, and assign the task to each node;

[0139] Step S4: construct a resource scheduling dynamic graph, set up a visualization interface, and arrange the resource scheduling dynamic graph on the visualization interface to display resource scheduling.

[0140] The adaptive resource scheduling system for improving cluster reconstruction efficiency is applied to an adaptive resource scheduling method for improving cluster reconstruction efficiency, including:

[0141] Data collection module: monitors the resource usage data of each node in the cluster in real time through the monitoring agent, and obtains the health status of the node through the heartbeat mechanism;

[0142] Task processing module: collects input task requests, predicts the resource requirements of each task, performs preliminary queue processing on the tasks, and obtains the task queue group;

[0143] Resource scheduling module: sets resource competition constraints based on the resource usage data and health status of the task queue group and each node, uses the optimization algorithm to obtain the dynamic resource scheduling corresponding to each task, and assigns the task to each node;

[0144] Visual display module: used to interact with users and increase the user-friendly interface through multi-level visualization of data by users;

[0145] The modules are connected to each other via wired and / or wireless means.

[0146] In addition, according to the implementation of the present application, the process described in the attached figure of an adaptive resource scheduling system for improving cluster reconstruction efficiency can be implemented as a computer software program. For example, the present application provides a non-transitory machine-readable storage medium, which stores machine-readable instructions, and the machine-readable instructions can be executed by a processor to execute instructions corresponding to the method steps provided in the present application. Of course, the architecture shown in the attached figure of an adaptive resource scheduling system for improving cluster reconstruction efficiency is only exemplary. When implementing different devices, adaptive selection or adjustment can be performed according to actual needs.

[0147] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and thresholds in the formula are set by technicians in this field according to actual conditions.

[0148] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technical users in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. An adaptive resource scheduling method for improving cluster reconstruction efficiency, characterized in that: The adaptive resource scheduling method for improving cluster reconstruction efficiency includes: Step S1: Use the monitoring agent to monitor the resource usage data of each node in the cluster in real time, and obtain the health status of the node through the heartbeat mechanism; Step S2: Collect input task requests, predict the resource requirements of each task, perform preliminary queue processing on the tasks, and obtain a task queue group; Step S3: Set resource competition constraints according to the resource usage data and health status of the task queue group and each node, use the optimization algorithm to obtain the dynamic resource scheduling corresponding to each task, and assign the task to each node; Step S4: construct a resource scheduling dynamic graph, set up a visualization interface, and arrange the resource scheduling dynamic graph on the visualization interface to display resource scheduling.

2. The adaptive resource scheduling method for improving cluster reconstruction efficiency according to claim 1, characterized in that: The resource usage data includes CPU usage, memory usage, disk I / O and network bandwidth usage; Health status includes normal state and abnormal state; Deploy lightweight monitoring agents in parallel on each node, and the monitoring agents collect node resource usage in real time; Within the preset time t, the sampling frequency fs is set, and then the sampling length Cy=t×fs is calculated; the collected resource usage data is normalized and standardized; The collected resource usage data and health status are transmitted to the central monitoring server via the network for storage.

3. The adaptive resource scheduling method for improving cluster reconstruction efficiency according to claim 2, characterized in that: The method for obtaining the health status of a node through a heartbeat mechanism includes: Step Q1: Get the heartbeat interval of each node. For each node, when each heartbeat interval arrives, the monitoring agent sends a heartbeat request to the monitored node. The monitored node receives the heartbeat request and performs a corresponding action. After self-checking according to the corresponding action, it generates a heartbeat response and transmits the heartbeat response back to the monitoring agent. Among them, the heartbeat request includes network status check, service status check, application running status check and database connection status check, and the heartbeat response includes health status and error information; Step Q12: After receiving the heartbeat response, the monitoring agent analyzes it. If the heartbeat response indicates that the monitored node is in a normal state, the monitoring agent records and updates the health state, and sends the heartbeat request again after the heartbeat interval. If the heartbeat response indicates that the monitored node is in an abnormal state, the monitoring agent marks the node as abnormal and sends an alarm. Step Q3: If the monitoring agent does not receive a heartbeat response from the monitored node within the specified time, it will resend the heartbeat request until it receives a heartbeat response or reaches the limit condition. If it receives a heartbeat response, it will be processed according to step Q2. If it reaches the limit condition, it will mark the node as disconnected and send an alarm. The limit condition is that the heartbeat request resent within the heartbeat interval does not exceed re times; Step Q4: Collect the health status of all nodes and display them in summary.

4. The adaptive resource scheduling method for improving cluster reconstruction efficiency according to claim 3, characterized in that: The specific method for obtaining the heartbeat interval of each node includes: For each node, set the initial interval to represents the initial interval of the qth node, and sets the update of the heartbeat interval to in, It represents the iteratively updated heartbeat interval of the qth node when it sends the heartbeat request for the rth time. It represents the heartbeat interval that the qth node iteratively updates when it sends a heartbeat request for the r-1th time. It represents the speed of iterative update of the qth node at the rth time, represents the inertia weight of the qth node at the rth iteration update, ∈(r) represents the random adjustment item at the rth iteration update; speed Obtained through resource usage data and historical health status of nodes, in, represents the speed of iterative update of the qth node at the r-1th time, α1, α2, α3, α4 and α5 represent weight coefficients, S_cpu r 、S_mem r 、S_dis r 、S_net r and G_Z r Indicates the CPU usage, memory usage, disk I / O, network bandwidth usage, and number of node failures of the node during the rth iteration update, S_cpu r-1 、S_mem r-1 、S_dis r-1 、S_net r-1 and G_Z r -1 Indicates the CPU usage, memory usage, disk I / O, network bandwidth usage, and number of node failures of the node during the r-1th iteration update. and Indicates the node's standard CPU usage, memory usage, disk I / O, network bandwidth usage, and number of node failures; Inertia Weight Calculate based on the changing trend of heartbeat intervals, Indicates the heartbeat interval that is iteratively updated when the qth node sends a heartbeat request for the r-2th time.

5. The adaptive resource scheduling method for improving cluster reconstruction efficiency according to claim 4, characterized in that: The method for queue processing of tasks comprises: Extract historical task request data and use machine learning models to predict the resource requirements of each task; Set initial priorities for each mission based on confidentiality, urgency, and resource requirements Use the decay mechanism to update the priority of each task in, represents the initial priority level of the i-th task, Indicates the priority level of the i-th task at the j-th time, represents the priority level of the i-th task at the j-1th time, γ1, γ2 and γ3 represent the proportion coefficients, and Bm i Indicates the confidentiality level of the i-th task, Jm i Indicates the urgency of the i-th task, Zm i represents the resource requirement of the ith task, represents the growth coefficient, T i Indicates the waiting time of the i-th task not being processed; Calculate the priority of each task in turn and arrange them in order from high to low.

6. The adaptive resource scheduling method for improving cluster reconstruction efficiency according to claim 5, characterized in that: The method of using a machine learning model to predict the resource requirements of each task includes: Collect historical task feature data, including task type, task size, task complexity, and the resource requirements of the corresponding tasks to form a sample set; Normalize and standardize the data of the sample set before preprocessing; Use the sample set to train the machine learning model, which is a neural network or deep learning model. Use the task feature data as the input of the model, and the resource demand as the output of the model. Set the mean square error as the loss function. When the loss function value is the minimum or no longer changes, the training stops and the trained machine learning model is obtained. Input the task feature data of each current task into the trained machine learning model to obtain the resource requirements of each task.

7. The adaptive resource scheduling method for improving cluster reconstruction efficiency according to claim 6, characterized in that: The method for setting the resource competition constraint includes: The resource competition constraints are set to include a first constraint, a second constraint and a third constraint, and the priority level is that the first constraint is greater than the second constraint, and the second constraint is greater than the third constraint; The first constraint is that the nodes to which tasks are assigned are all in normal state, and are recorded as assignable nodes; The second constraint is that the resource usage data of the allocatable nodes are all below the upper threshold; The third constraint is that tasks of the same priority are allocated in descending order of resource requirements.

8. The adaptive resource scheduling method for improving cluster reconstruction efficiency according to claim 7, characterized in that: The method of using the optimization algorithm to obtain the dynamic resource scheduling corresponding to each task includes: Under the condition of satisfying resource competition constraints, an optimization algorithm is used to allocate nodes; Step D1: The optimization algorithm is based on the improved particle swarm algorithm. The number of nodes that can be assigned tasks is obtained. The same number of particle swarms is set according to the number of nodes, which is recorded as behavior particle swarms. Each task generates a behavior particle swarm. Step D2: Set the starting position of the particle to the task queue group and the end position to the node where the task is stored. Use the fitness function to evaluate the quality of the path. The fitness function Among them, H_zy represents the remaining resource usage data of the node, R_zy represents the resource demand of the task, Kx represents the path cost, Yc represents the delay time, μ1 and μ2 represent the weight coefficients; Step D3: With the goal of minimizing the fitness function, the tasks of the same priority level are calculated in turn according to the task queue group. For each task, the node with the smallest fitness function is selected as the allocation node, and the task is allocated to the corresponding node; Step D4: Update the task queue group, repeat steps D1 to D3, and sequentially allocate nodes to the continuously updated task requests.

9. The adaptive resource scheduling method for improving cluster reconstruction efficiency according to claim 8, characterized in that: The method for constructing the resource scheduling dynamic graph includes: Resource scheduling dynamic graphs include task graphs and task allocation graphs; The task chart includes establishing a blank two-dimensional coordinate chart, taking the time point as the horizontal coordinate and the task priority level as the vertical coordinate, and filling the task request at each time point into the corresponding position according to the task priority level; The task allocation chart includes establishing a blank plane, simulating all task allocation nodes into first solid points, simulating the location where the task request occurs into second solid points, representing the first solid points and the second solid points with different colors, and filling them into the blank plane according to the actual positions, obtaining the nodes corresponding to each task, that is, the first solid points corresponding to each second solid point, connecting them through directed edges to display the corresponding allocation of tasks and nodes.

10. The adaptive resource scheduling method for improving cluster reconstruction efficiency according to claim 9, characterized in that: The visualization interface is used to interact with the user and provide the user with multi-level visualization of the data.

Images