Distributed cluster resource autonomous scheduling method based on DSACO

Abstract

The present invention discloses a distributed cluster resource autonomous scheduling method based on DSACO, which mainly includes: real-time monitoring of the status of each node and network topology information in the cluster system, and dynamic tracking of the operation of the distributed cluster; secondly, based on the multi-objective planning model and the group intelligence optimization idea, a distributed resource autonomous scheduling model is constructed; based on the resource autonomous allocation model, a multi-stage resource allocation algorithm (DSACO) is constructed to fairly select a task set, conduct a decentralized search for different tasks, further optimize the preliminary allocation results, and determine a better resource allocation scheme; finally, judgment and processing are performed based on the QoS evaluation index to enhance the cluster reliability under high load conditions. The present invention optimizes resource allocation from multiple dimensions such as resource allocation fairness, resource utilization, and comprehensive load, which can improve the rapid batch allocation capability of distributed cluster resources in large systems and improve the service quality under high load conditions.

Description

Technical Field

[0001] The present invention belongs to the technical field of high reliability and high availability, and in particular to a distributed cluster resource autonomous scheduling method based on DSACO. Background Art

[0002] In recent years, with the rapid development of the internet, distributed computing has become increasingly widespread, finding widespread application in various industries, including finance, healthcare, e-commerce, social media, and the Internet of Things. Distributed clusters, by distributing computing tasks and data storage across multiple nodes, can effectively improve data processing capabilities, system reliability, and user experience. For example, when building large websites, distributed computing can significantly improve system response speed and stability by distributing user requests across multiple servers through load balancing technology. Distributed clusters also play a key role in big data processing, cloud computing, and edge computing, supporting complex data analysis, real-time data processing, and a wide range of service deployments.

[0003] However, existing research has largely focused on data consistency, load balancing, and fault tolerance, neglecting the optimization of resource allocation in distributed environments. Resource allocation is crucial in distributed clusters because it directly impacts system performance, efficiency, and cost. Improper resource allocation can lead to resource waste, task congestion, and response delays, reducing overall system efficiency and user experience. Resource allocation is particularly challenging in scenarios with dynamically changing resource demands, such as high-concurrency access and real-time data processing.

[0004] For example, cloud service providers dynamically provide virtual machine services to users. This process is achieved by deploying virtual machine monitors on physical machines. Placing different types of virtual machines on a single physical machine requires different degrees of partitioning of the physical machine's resources. If the resource allocation of thousands of virtual machines cannot optimally partition the physical machine resources, then the data center will need more physical machines to meet the needs of cloud customers. Adding more physical machines to the data center will lead to higher energy consumption, hardware costs, and maintenance costs, while causing serious resource waste. Therefore, it is crucial to study fair and effective resource allocation strategies. In large-scale cloud computing systems, effective and fair resource allocation is particularly challenging. This is because: (1) there are many types of resources, such as computing power, storage space, and network bandwidth; (2) workloads have diverse and rapidly changing demands for these resources; (3) server resource capacity is heterogeneous, and the performance and configuration of different servers vary; (4) workloads may have other restrictions on the use of servers, such as geographical location, network latency, etc. These challenges make it very complicated to achieve efficient and fair resource allocation in practical applications.

[0005] Researchers have proposed a variety of optimization algorithms and strategies for different types of distributed clusters and application scenarios. Their main research directions fall into traditional mathematical optimization models, swarm intelligence optimization strategies, and resource allocation strategies integrated with machine learning. These methods can improve resource scheduling and task allocation balance to a certain extent, while also enabling more comprehensive resource management and monitoring. Typical work includes: M. Wang et al. proposed the Hadoop resource scheduling algorithm TMSA based on task allocation. This algorithm calculates and analyzes the degree of adaptability between each task and each node, uses a prediction algorithm to estimate the final task completion time, and compares the prediction results to obtain a final allocation plan, thereby optimizing the overall task completion time. However, as the number of tasks increases, the algorithm's computational complexity increases significantly, leading to unstable performance. HU et al. analyzed node real-time performance and configuration and proposed a cluster resource allocation algorithm that adapts node real-time performance to the node's real-time performance. This algorithm analyzes dynamic and static resource weights and selects nodes based on the task type, enabling node real-time performance to adapt to the task type. Nodes with the best real-time performance are assigned to tasks, thereby improving the overall performance of the cluster system. However, the resource weights in this paper are fixed, making it unsuitable for complex clusters. Mirzaei et al. proposed a new architecture for offloading jobs and allocating resources in the Internet of Things. This architecture uses the Moth to Flame (MFO) algorithm combined with an adversarial learning (OBL) optimization approach for resource allocation, avoiding system load imbalance and ensuring data accuracy. However, due to the high sensitivity of the algorithm parameters, performance instability may occur in complex cluster scenarios. Muhammad et al. proposed a method that combines a load-balancing task scheduler with a machine learning-based device predictor. The predictor predicts execution times on CPU and GPU devices, with devices with the shortest predicted execution times being considered suitable for a particular task. A work-stealing-based task scheduler was designed as part of the solution to optimize the predictions. However, due to the high node resource consumption and potential conflicts in machine learning, the second-stage task scheduling becomes complex, making it unsuitable for clusters with large task volumes and complex operations.

[0006] On the other hand, in order to address the issue of resource allocation fairness in cloud computing environments, DRF (Dominant Resource Fairness) was proposed and promoted as a fair allocation strategy. DRF aims to balance the global dominant share of each user in multi-resource application scenarios. This definition is based on the highest proportion of any resource allocated to the user in the resource pool. DRF not only significantly improves the fairness of resource allocation, but its extended heterogeneous DRF (Heterogeneous Dominant Resource Fairness, HDRF) has been introduced as an advanced multi-resource scheduling algorithm to effectively prevent starvation of jobs in different data centers due to uneven resource allocation. However, there is still room for further improvement in the optimization of resource allocation rate and the refinement of cluster load management.

[0007] In summary, although good research results have been achieved in the scheduling of distributed cluster resources, the following issues still need further research: (1) Insufficient balance of multi-dimensional performance indicators such as resource utilization, fairness, and cluster load; (2) Limited adaptability to complex dynamic environments; (3) Algorithm execution efficiency and resource usage need to be optimized. Summary of the Invention

[0008] The present invention aims to address the aforementioned problems in the existing technologies by providing a distributed cluster resource autonomous scheduling method based on DSACO. This invention focuses on optimizing the balance of multi-dimensional performance indicators, improving adaptability to complex dynamic environments, and optimizing algorithm execution efficiency and resource utilization, aiming to contribute new solutions and ideas to this field.

[0009] The technical solution to achieve the purpose of the present invention is as follows: On the one hand, a distributed cluster resource autonomous scheduling method based on DSACO is provided, the method comprising:

[0010] Step 1: Building a system information repository RES based on node configuration information, node dynamic information, and network information, while simultaneously monitoring load status information on each node in the distributed cluster and link information between nodes in real time, and storing the information in the system information repository RES;

[0011] Step 2: Evaluate the server nodes from the aspects of computing capacity, load, and resource utilization, calculate the comprehensive load and resource utilization of each node, and save them in the system information repository RES;

[0012] Step 3: Based on the heterogeneity of nodes and user tasks in the server cluster, the resource scheduling problem is modeled as a multi-objective optimization problem, and a distributed resource autonomous allocation model is constructed;

[0013] Step 4: Based on the distributed resource autonomous allocation model, and taking into account the differences in priority and resource requirements of user tasks, a fair allocation algorithm with priority iteration is used to construct a set of allocable tasks from all tasks within the available amount of cluster resources.

[0014] Step 5: Based on the set of allocatable tasks and cluster node status information, a cluster resource decentralized search and competition optimization algorithm (DSACO) is constructed, so that each task searches for a server node with high adaptability based on its own resource demand characteristics, and determines a preliminary resource allocation plan.

[0015] Step 6: Analyze the expected load changes of nodes, execute a competitive optimization algorithm on nodes with expected load differences greater than a preset standard, and determine a resource allocation plan. The competitive optimization algorithm includes: using a consistent hashing algorithm to randomly assign low-priority tasks to low-load nodes, and then performing a combinatorial optimization of the random assignment results based on a swarm intelligence optimization algorithm.

[0016] In step 7, the expected load after resource allocation is evaluated using the comprehensive load and node pre-assigned task characteristics. If the expected load of the node does not exceed the set threshold, resource allocation is performed; otherwise, considering the fairness and priority of resource allocation, an elimination strategy is used for further processing to reduce the number of tasks to be executed to reduce the expected load.

[0017] In another aspect, a distributed cluster resource autonomous scheduling system based on DSACO is provided, the system comprising:

[0018] The first module is used to build a system information repository RES based on node configuration information, node dynamic information and network information, and monitor the load status information of each node in the distributed cluster and the link information between nodes in real time, and store them in the system information repository RES;

[0019] The second module is used to evaluate the server nodes from the aspects of computing capacity, load, and resource utilization, calculate the comprehensive load and resource utilization of each node, and save them in the system information repository RES;

[0020] The third module is used to model the resource scheduling problem as a multi-objective optimization problem based on the heterogeneity of nodes and user tasks in the server cluster, and to build a distributed resource autonomous allocation model;

[0021] The fourth module is used to construct an allocatable task set from all tasks within the available amount of cluster resources based on the distributed resource autonomous allocation model and according to the differences in priority and resource requirements of user tasks through a fair allocation algorithm with priority iteration;

[0022] The fifth module is used to build a cluster resource decentralized search and competition optimization algorithm (DSACO) based on the set of allocatable tasks and cluster node status information. This algorithm enables each task to search for a server node with high adaptability based on its own resource demand characteristics and determine a preliminary resource allocation plan.

[0023] The sixth module is used to analyze the expected load changes of nodes, execute a competitive optimization algorithm for nodes with expected load differences greater than a preset standard, and determine a resource allocation plan. The competitive optimization algorithm includes: using a consistent hashing algorithm to randomly assign low-priority tasks to low-load nodes, and further performing combinatorial optimization on the random assignment results based on a swarm intelligence optimization algorithm;

[0024] The seventh module is used to evaluate the expected load after resource allocation using the comprehensive load and node pre-assigned task characteristics. If the expected load of the node does not exceed the set threshold, resource allocation is performed; otherwise, considering the fairness and priority of resource allocation, an elimination strategy is used for further processing to reduce the number of tasks to be executed to reduce the expected load.

[0025] Compared with the prior art, the present invention has the following significant advantages:

[0026] (1) Taking into account the fairness of task allocation, resource utilization and load multiple indicators, the overall performance and user experience of the cluster system are improved.

[0027] (2) A priority-iterative dominant resource fair allocation method is proposed, which comprehensively considers three aspects: task priority, task submission time, and the amount of allocated resources to ensure the fairness of resource allocation.

[0028] (3) A resource autonomous scheduling model is constructed to solve the resource allocation problem. A method for constructing a resource search space is given, which can comprehensively analyze the distribution characteristics of cluster resources in real time, making the resource allocation plan more complete and accurate.

[0029] (4) Based on the characteristics of resources in heterogeneous clusters, a multi-stage optimization algorithm, the decentralized search and competition optimization algorithm DSACO, is proposed. This intelligent optimization algorithm has a stronger ability to search for the optimal solution, solves the resource competition problem of local optimal solutions, and shortens the processing time of searching for the optimal resource allocation solution.

[0030] The present invention is further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is the structural diagram of the distributed cluster resource autonomous scheduling method based on DSACO.

[0032] Figure 2 Flowchart of the distributed cluster resource autonomous scheduling method based on DSACO.

[0033] Figure 3 A comparison chart of cluster node resource utilization experimental results in one embodiment.

[0034] Figure 4 This is a task completion time comparison chart in one embodiment. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0036] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indications will also change accordingly.

[0037] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features specified as "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but this must be based on the fact that ordinary technicians in this field can implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0038] In traditional cluster service systems, resource allocation algorithms pay little attention to resource utilization, resulting in low allocation efficiency. Figure 1 and Figure 2 The present invention provides a distributed cluster resource autonomous scheduling method based on DSACO, the method comprising the following steps:

[0039] Step 1: Building a system information repository RES based on node configuration information, node dynamic information, and network information, while simultaneously monitoring load status information on each node in the distributed cluster and link information between nodes in real time, and storing the information in the system information repository RES;

[0040] Step 2: Evaluate the server nodes from the aspects of computing capacity, load, and resource utilization, calculate the comprehensive load and resource utilization of each node, and save them in the system information repository RES;

[0041] Step 3: Based on the heterogeneity of nodes and user tasks in the server cluster, the resource scheduling problem is modeled as a multi-objective optimization problem, and a distributed resource autonomous allocation model is constructed;

[0042] Step 4: Based on the distributed resource autonomous allocation model, and taking into account the differences in priority and resource requirements of user tasks, a fair allocation algorithm with priority iteration is used to construct a set of allocable tasks from all tasks within the available amount of cluster resources.

[0043] Step 5: Based on the set of allocatable tasks and cluster node status information, a cluster resource decentralized search and competitive optimization algorithm (DSACO) is constructed. This algorithm enables each task to search for a server node with high adaptability based on its own resource demand characteristics and determine a preliminary resource allocation plan.

[0044] Step 6: Analyze the expected load changes of nodes, execute a competitive optimization algorithm on nodes with expected load differences greater than a preset standard, and determine a resource allocation plan. The competitive optimization algorithm includes: using a consistent hashing algorithm to randomly assign low-priority tasks to low-load nodes, and then performing a combinatorial optimization of the random assignment results based on a swarm intelligence optimization algorithm.

[0045] In step 7, the expected load after resource allocation is evaluated using the comprehensive load and node pre-assigned task characteristics. If the expected load of the node does not exceed the set threshold, resource allocation is performed; otherwise, considering the fairness and priority of resource allocation, an elimination strategy is used for further processing to reduce the number of tasks to be executed to reduce the expected load.

[0046] Furthermore, in one embodiment, the step 1 of constructing the system information repository RES based on the node configuration information, node dynamic information and network information is as follows:

[0047] RES=(RES static ,RES dynamic ,RES network #(1)

[0048] Among them, RES static It is the static resource of the cluster, i.e. the basic configuration information of the server nodes and the network. When a new service node joins the cluster, it collects resource information, which remains unchanged during the subsequent system operation. The detailed description of the static resource is as follows:

[0049] RES static =(CN,MS,DS,CF,DR,NB)#(2)

[0050] Where CN is the number of CPU cores of the server node, MS is the memory size of the node, disk size DS, CF is the clock frequency, DR is the disk IO rate, and NB is the network bandwidth;

[0051] Among them, RES dynamic is the real-time status information of the node, which is collected again each time resource allocation is performed to monitor the status of cluster nodes in real time. The detailed description of the resources is as follows:

[0052] RES dynamic =(CN use ,MS use ,DS use ,NB use )#(3)

[0053] Where, CN use Indicates CPU usage, MS use Indicates memory usage, NB use is the node bandwidth utilization, DS use For disk usage;

[0054] Among them, RES networ k is the network state parameter of the node, which is described in detail as formula (4):

[0055] RES network =(NDE avg ,NT avg ,NL avg )#(4)

[0056] Where NT avg is the average throughput of the cluster where the node is located, NL avg is the average packet loss rate of the cluster where the node is located, NDE avg is the average transmission delay.

[0057] Furthermore, in one embodiment, in step 2, the comprehensive load and resource utilization of each node are calculated based on the collected node load status information and computing capacity;

[0058] The calculation formula of the comprehensive load is as follows:

[0059]

[0060] Where NWL i is the comprehensive load of the i-th node. According to the characteristics of heterogeneous cluster tasks, the load is divided into IO load NL i and calculate the load CL iThere are two parts: the network IO load is caused by complex computing-related tasks and is mainly determined by performance parameters such as network bandwidth, network latency, and network transmission rate; the computing load is caused by big data computing-related tasks and is determined by the number of CPU cores, memory size, clock frequency, and disk IO rate;

[0061] The calculation formula of resource utilization is shown in formula (6):

[0062]

[0063] Where, SRU i represents the resource utilization of the i-th server node, ω j represents the weight of resource j, SRC i ={SRC i1 ,SRC i2 …SRC im} represents the resource capacity vector of the i-th server node, where m is the number of resource types, SRC ij Indicates the total amount of resource j in the i-th node of the server, SRL i ={SRL i1 ,SRL i2 …SRL im} represents the available capacity of resources in server i, SRL ij represents the available capacity of resource j in server i.

[0064] Furthermore, in one embodiment, step 3 specifically includes:

[0065] Step 3.1, establish cluster system model MODEL system :

[0066] MODEL system =(S,R,U,W)#(7)

[0067] Among them, S represents the cluster node set, R corresponds to the node resource set, U represents the system user set, and W represents the user's task set;

[0068] Step 3.2: Establish resource scheduling optimization objectives and constraints:

[0069] The problem to be solved in resource allocation is to reasonably allocate cluster resources to user tasks and realize resource scheduling, which is essentially to establish a mapping relationship between tasks and cluster nodes. For this purpose, the present invention proposes the following optimization goals: (1) task allocation fairness; (2) resource utilization; (3) load balancing; and (4) algorithm execution time.

[0070] (1) Resource allocation fairness, used to determine the balance of resource allocation to users and improve user experience. The calculation formula is:

[0071]

[0072] Among them, f fairness The fairness of resource allocation is evaluated by the variance of the maximum resource allocation amount of the user, UTR u represents the maximum total amount of resource allocation for user u, Indicates the average of the total amount of resource allocation, n represents the number of users in the cluster, URV u ={URV u1 ,URV u2 ,…,URV um} represents the resource demand vector of user u, where URV uj represents the demand of user u for resource j, UTN u represents the number of tasks of user u that has obtained resources, RL={RL1,RL2…RL m} represents the cluster resource availability vector, RL j represents the available amount of cluster resource j, and U is the user set;

[0073] Resource allocation should ensure that the total amount of allocated resources is less than the available amount of cluster resources. The judgment method is:

[0074]

[0075] (2) Cluster load balancing, used to evaluate load balancing and resource utilization balance, is calculated as follows:

[0076]

[0077] Among them, f loadbalance For cluster load balancing, represents the average load of the cluster, sn represents the number of server nodes, and is evaluated by the variance of the total load of each node;

[0078] (3) Cluster resource utilization, which is used to evaluate the rationality of resource allocation and further compensate for the problem of ineffective resource utilization caused by fairness. The calculation formula is:

[0079]

[0080] Among them, f usage is the cluster resource utilization;

[0081] (4) Resource scheduling time: used to evaluate the efficiency of algorithm execution, by monitoring the time interval from the start of each task scheduling to the determination of the mapping relationship and submission to the node, and determining the resource scheduling time f time The maximum time interval of the currently scheduled task is calculated as follows:

[0082] f time =max(w end -w start )w∈W#(12)

[0083] Where w start represents the start time of resource scheduling, i.e. the start time of task fair selection, w end Indicates the time when the mapping relationship is finally determined;

[0084] Step 3.3: Use the linear regression model to solve the multi-objective optimization problem of the resource scheduling process and establish a distributed resource autonomous allocation model:

[0085]

[0086] Where, ξ f ,ξ t ,ξ u ,ξ l Corresponding weights for different optimization objectives;

[0087] In order to unify the optimization objectives, the inverse of resource utilization is taken. Therefore, the smaller the values ​​of each optimization objective are, the better the allocation plan given by the resource scheduling method is. The optimization objective of the overall model is to minimize f.

[0088] Furthermore, in one embodiment, the fair allocation algorithm of priority iteration in step 4 is specifically as follows: when the total task resource demand is greater than the available cluster resource and all tasks cannot be allocated, a set of tasks that can currently be allocated is selected from all tasks.

[0089] Here, a fair allocation algorithm with priority iteration is designed to analyze the balance problem between user priority and fair allocation.

[0090] If only task priority is considered, it will cause low resource utilization. If only fairness is analyzed, some tasks may not be executed for a long time. Therefore, the present invention designs a priority-iterative task fair allocation algorithm. When the total task resource demand is greater than the available cluster resources and it is impossible to allocate all tasks, a set of tasks that can currently be allocated is selected from all tasks to ensure the fairness of resource allocation.

[0091] Furthermore, in one embodiment, step 4 specifically includes:

[0092] Step 4.1: Considering the user's task priority and the amount of resources allocated to the user, the minimum resource allocator is obtained by the following formula:

[0093]

[0094] Among them, UTN={UTN1,UTN2…UTN n}, UTN u Indicates the number of tasks that have resources allocated to user u, UTN u Satisfy:∑ u＝1..n UTN u *URV uj ≤RL j , That is, the amount of allocated resources is less than the available amount of cluster resources; URV u ={URV u1 ,URV u2 …URV um} represents the task resource requirement vector of user u, RL={RL1,RL2…RL m} represents the set of available resources in the cluster, priority u Indicates task priority; represents the user's dominant resource, Indicates that user u has demand; The calculation formula is:

[0095]

[0096] Step 4.2, allocating resources for the tasks of the minimum resource allocator;

[0097] Here, if there are multiple minimum resource allocators at the same time, the priority is determined and the minimum resource allocator with the highest priority is selected for allocation;

[0098] Step 4.3: Loop through steps 4.1 and 4.2 until the total resource requirements of the selected tasks reach the current available cluster resources (there are insufficient available resources when selecting the next task). Then, the selection process stops, and the allocable task set WSL is determined.

[0099]

[0100] Step 4.4: Increase the priority of currently unselected tasks outside the allocatable task set WSL to ensure that these tasks are selected first next time.

[0101] Furthermore, in one embodiment, step 5 specifically includes:

[0102] Phase 1: Determine decision variables and construct search space:

[0103] Step 5.1: Construct an algorithm search space based on the server resources and task characteristics. This includes:

[0104] Step 5.1.1: Analyze the allocatable task set WSL and determine the number of dimensions of the search space to be the number of resource requirement types in the task requirement set;

[0105] Step 5.1.2: Analyze the available resources of the cluster nodes and determine the boundary value of each dimension as the maximum available amount of cluster node resources;

[0106] Step 5.1.3, locate the node to the corresponding position in the search space according to the server node resource availability vector;

[0107] Step 5.2: Determine the initial location of the task based on the task resource demand vector and the available resources of the cluster nodes. The steps are as follows:

[0108] Step 5.2.1, calculate and mark the minimum available resources of each node;

[0109] Step 5.2.2: Analyze the dominant resource of the task and map the task to the node with the smallest available dominant resource.

[0110] Step 5.3: Generate the initial direction and speed of each task. The speed of task w is v and the direction is o. The total number of iterations of the search process is δ.

[0111] Phase 2: Determine the allocation results through decentralized search, including:

[0112] Step 5.4: Start iterative search, calculate the fitness value of the task at the current node according to the following formula, and update the historical optimal fitness value of the task;

[0113] FIT i =α T RUT i +α D RUD i +α L NWL i #(17)

[0114] Where FIT i Indicates the fitness value of the current task and the i-th node, RUT i Represents the resource utilization change after the i-th node allocates resources to the current user u's task w; NWL i is the comprehensive load of node i after the task is assigned to the node, and the load is defined as shown in formula (5); α T , α D and α L Represents the weight parameters of the three indicators respectively; RUT i Calculation formula:

[0115]

[0116] RUD i Represents the deviation of the current server node resource utilization from the global average resource utilization, RUD i The calculation formula is:

[0117]

[0118] Among them, SRU i` Indicates the resource utilization of the i-th node of the server;

[0119] Step 5.5, determine the moving direction of the next task in the search space;

[0120] The optimal search direction is determined based on the task resource demand vector, and a random interference vector is introduced to interfere to prevent the final search results from being too concentrated, thereby improving the global optimization capability. The search direction and speed of task w for user u at a certain moment are shown in the following formula:

[0121]

[0122] in, is the weight of the random vector, which is used to adjust the optimization ability and convergence speed. Random is the random interference vector, and O is the vector of speed and direction.

[0123] In step 5.6, the above iterative process is repeated until the final number of iterations is reached, and the fitness value of the current position is compared with the historical optimal position. The task is moved to the node with higher fitness, the allocation result is confirmed, and the preliminary resource allocation is completed.

[0124] Furthermore, in one embodiment, the analysis of the expected load change of the nodes in step 6 and the execution of the competition optimization algorithm for the nodes with large expected load differences specifically include:

[0125] If multiple tasks are eventually optimized for the same node, resources are directly allocated if the node has sufficient resources; otherwise, a competitive optimization algorithm is executed.

[0126] In some embodiments herein, step 6 specifically includes:

[0127] Step 6.1: For each server node with task conflicts, select the high-priority task for allocation based on task priority, and then integrate the unselected tasks into the task set to be optimized WCL. Task conflicts refer to tasks where the resource requirements are greater than the available resources of the node.

[0128] In step 6.2, the consistent hashing algorithm is used for random allocation. All nodes are connected into a hash ring, and tasks in the WCL are randomly assigned to nodes. If a node has insufficient resources, it moves to the next node in the specified direction. After a certain number of moves, if there are still tasks that cannot find a corresponding node, the task is removed and the priority is iterated to participate in the next round of resource allocation.

[0129] Step 6.3: Optimize the random assignment result of step 6.2. Specifically, combine the idea of ​​the Black Widow Optimization Algorithm and compare whether swapping the two tasks can obtain a better fitness value. If the fitness is better, then swap it. If it cannot achieve a better effect, then eliminate it and obtain a further assignment plan. The optimization steps are as follows:

[0130] In step 6.3.1, each server is abstracted as an individual in the Black Widow population, and the optimization rate pr, elimination rate cr, optimization times on, optimization threshold ot, and the number of servers participating in the optimization N are defined in the algorithm.

[0131] The cluster nodes are sorted according to the resource utilization rate, and N×pr individuals with low resource utilization in the cluster are selected for optimization based on the optimization rate pr. The individuals participating in the optimization are randomly grouped into teams for reproduction. The reproduction process is shown in the following formula:

[0132]

[0133] Among them, S1 and S2 are the selected node pairs to be optimized, S`1 and S`2 are the optimized node pairs, and α is a randomly generated probability array of auxiliary reproduction;

[0134] Step 6.3.2: Calculate the fitness of the newly bred individuals according to formula (17), reorder the N individuals, select N × cr individuals with high fitness in the population based on the elimination rate cr, and optimize the remaining individuals.

[0135] In step 6.3.3, if the upper limit of the number of combined optimizations is reached or the optimization effect cannot reach the optimization threshold, the algorithm terminates and the optimized resource allocation plan is obtained.

[0136] Since the allocation result in step 6.3 may not be completely in line with expectations in actual operation, there may be a situation where the actual operation result exceeds the expected result and causes the server node to overload and crash, so further redundant fault tolerance is required for the result.

[0137] Furthermore, in one embodiment, step 7 specifically includes:

[0138] For the resource allocation scheme obtained in step 6, evaluate it according to equations (5) and (6), calculate the expected comprehensive load and resource utilization of each node, and compare whether the expected result exceeds the node load threshold. If NWL i If the load threshold is exceeded, some tasks will be eliminated to ensure the normal operation of the cluster;

[0139] Among them, the elimination strategy is as follows:

[0140]

[0141] That is, the priority of the task set on the slave node w , resource allocation fairness f fairness The fitness value FIT of the task at the node is comprehensively evaluated, and the task with the smallest evaluation result is eliminated.

[0142] In one embodiment, a distributed cluster resource autonomous scheduling system based on DSACO is provided, the system comprising:

[0143] The first module is used to build a system information repository RES based on node configuration information, node dynamic information and network information, and monitor the load status information of each node in the distributed cluster and the link information between nodes in real time, and store them in the system information repository RES;

[0144] The second module is used to evaluate the server nodes from the aspects of computing capacity, load, and resource utilization, calculate the comprehensive load and resource utilization of each node, and save them in the system information repository RES;

[0145] The third module is used to model the resource scheduling problem as a multi-objective optimization problem based on the heterogeneity of nodes and user tasks in the server cluster, and to build a distributed resource autonomous allocation model;

[0146] The fourth module is used to construct an allocatable task set from all tasks within the available amount of cluster resources based on the distributed resource autonomous allocation model and according to the differences in priority and resource requirements of user tasks through a fair allocation algorithm with priority iteration;

[0147] The fifth module is used to build a cluster resource decentralized search and competition optimization algorithm (DSACO) based on the set of allocatable tasks and cluster node status information. This algorithm enables each task to search for a server node with high adaptability based on its own resource demand characteristics and determine a preliminary resource allocation plan.

[0148] The sixth module is used to analyze the expected load changes of nodes, execute a competitive optimization algorithm for nodes with expected load differences greater than a preset standard, and determine a resource allocation plan. The competitive optimization algorithm includes: using a consistent hashing algorithm to randomly assign low-priority tasks to low-load nodes, and further performing combinatorial optimization on the random assignment results based on a swarm intelligence optimization algorithm;

[0149] The seventh module is used to evaluate the expected load after resource allocation using the comprehensive load and node pre-assigned task characteristics. If the expected load of the node does not exceed the set threshold, resource allocation is performed; otherwise, considering the fairness and priority of resource allocation, an elimination strategy is used for further processing to reduce the number of tasks to be executed to reduce the expected load.

[0150] For the specific definition of the distributed cluster resource autonomous scheduling system based on DSACO, please refer to the definition of the distributed cluster resource autonomous scheduling method based on DSACO above, which will not be repeated here. The various modules in the above-mentioned distributed cluster resource autonomous scheduling method based on DSACO can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0151] In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following is achieved:

[0152] Step 1: Building a system information repository RES based on node configuration information, node dynamic information, and network information, while simultaneously monitoring load status information on each node in the distributed cluster and link information between nodes in real time, and storing the information in the system information repository RES;

[0153] Step 2: Evaluate the server nodes from the aspects of computing capacity, load, and resource utilization, calculate the comprehensive load and resource utilization of each node, and save them in the system information repository RES;

[0154] Step 3: Based on the heterogeneity of nodes and user tasks in the server cluster, the resource scheduling problem is modeled as a multi-objective optimization problem, and a distributed resource autonomous allocation model is constructed;

[0155] Step 4: Based on the distributed resource autonomous allocation model, and taking into account the differences in priority and resource requirements of user tasks, a fair allocation algorithm with priority iteration is used to construct a set of allocable tasks from all tasks within the available amount of cluster resources.

[0156] Step 5: Based on the set of allocatable tasks and cluster node status information, a cluster resource decentralized search and competition optimization algorithm (DSACO) is constructed, so that each task searches for a server node with high adaptability based on its own resource demand characteristics, and determines a preliminary resource allocation plan.

[0157] Step 6: Analyze the expected load changes of nodes, execute a competitive optimization algorithm on nodes with expected load differences greater than a preset standard, and determine a resource allocation plan. The competitive optimization algorithm includes: using a consistent hashing algorithm to randomly assign low-priority tasks to low-load nodes, and then performing a combinatorial optimization of the random assignment results based on a swarm intelligence optimization algorithm.

[0158] In step 7, the expected load after resource allocation is evaluated using the comprehensive load and node pre-assigned task characteristics. If the expected load of the node does not exceed the set threshold, resource allocation is performed; otherwise, considering the fairness and priority of resource allocation, an elimination strategy is used for further processing to reduce the number of tasks to be executed to reduce the expected load.

[0159] For the specific limitations of each step, please refer to the limitations of the distributed cluster resource autonomous scheduling method based on DSACO above, which will not be repeated here.

[0160] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements:

[0161] Step 1: Building a system information repository RES based on node configuration information, node dynamic information, and network information, while simultaneously monitoring load status information on each node in the distributed cluster and link information between nodes in real time, and storing the information in the system information repository RES;

[0162] Step 2: Evaluate the server nodes from the aspects of computing capacity, load, and resource utilization, calculate the comprehensive load and resource utilization of each node, and save them in the system information repository RES;

[0163] Step 3: Based on the heterogeneity of nodes and user tasks in the server cluster, the resource scheduling problem is modeled as a multi-objective optimization problem, and a distributed resource autonomous allocation model is constructed;

[0164] Step 4: Based on the distributed resource autonomous allocation model, and taking into account the differences in priority and resource requirements of user tasks, a fair allocation algorithm with priority iteration is used to construct a set of allocable tasks from all tasks within the available amount of cluster resources.

[0165] Step 5: Based on the set of allocatable tasks and cluster node status information, a cluster resource decentralized search and competition optimization algorithm (DSACO) is constructed, so that each task searches for a server node with high adaptability based on its own resource demand characteristics, and determines a preliminary resource allocation plan.

[0166] Step 6: Analyze the expected load changes of nodes, execute a competitive optimization algorithm on nodes with expected load differences greater than a preset standard, and determine a resource allocation plan. The competitive optimization algorithm includes: using a consistent hashing algorithm to randomly assign low-priority tasks to low-load nodes, and then performing a combinatorial optimization of the random assignment results based on a swarm intelligence optimization algorithm.

[0167] In step 7, the expected load after resource allocation is evaluated using the comprehensive load and node pre-assigned task characteristics. If the expected load of the node does not exceed the set threshold, resource allocation is performed; otherwise, considering the fairness and priority of resource allocation, an elimination strategy is used for further processing to reduce the number of tasks to be executed to reduce the expected load.

[0168] For the specific limitations of each step, please refer to the limitations of the distributed cluster resource autonomous scheduling method based on DSACO above, which will not be repeated here.

[0169] As a specific example, in one of the embodiments, the present invention is further verified and explained in detail. A virtual machine cluster is built to compare the performance of the DSACO algorithm of the present invention with the Round Robin algorithm, the ATNPA algorithm, and the multi-objective particle swarm optimization algorithm (MOPSO). The MOPSO algorithm adopts a decomposition method, assigning an optimization sub-problem to each particle, and adopts a novel speed update strategy to accelerate the convergence speed. The ATNPA algorithm is a resource allocation algorithm proposed by Hu et al. that takes dynamic and static factors into consideration and can optimize cluster resource utilization and load balancing.

[0170] The distributed heterogeneous cluster system set up in the experiment consists of one central node and ten nodes with four different resource capacities. In a distributed cluster system, the central node monitors the status of other nodes in the cluster, analyzes user task requirements, and allocates resources. This example does not consider the task parsing steps. Based on task requirements such as file transfer and function computation, this example designs three types of tasks: I / O-intensive, CPU-intensive, and mixed-intensive.

[0171] This example uses the average of response time, resource utilization, and cluster load balancing, measured over ten actual runs, as the initial quality of service metric tuple for each resource allocation. The optimization and elimination rates in the DSACO algorithm are set to 0.6 and 0.1, respectively, with 60 optimizations and a threshold of 0.1 to ensure a stable population.

[0172] Figure 3 The change of cluster resource utilization as the number of services increases is shown. As the number of tasks increases, the resource occupancy of the four algorithms changes accordingly. From the experimental results, it can be seen that when the number of tasks is small, the resource utilization of each algorithm is equal due to the sufficient available resources in the cluster. However, as the number of tasks continues to increase, the resource utilization of the traditional polling algorithm decreases because polling allocation will cause a certain resource of the node to be exhausted while the utilization of other resources is very low. Comparing the MOPSO and ATNPA algorithms, it can be seen that the resource allocation strategy proposed in this invention is similar to the heuristic algorithm in terms of resource utilization. This is mainly because the present invention takes into account task fairness and the selected tasks have a certain impact on resource utilization. At the same time, when the expected load is too high, the elimination strategy is used to reduce tasks to ensure the reliability of the cluster.

[0173] Figure 4 The graph shows how task completion time changes as the number of tasks increases. As can be seen from the figure, compared to other intelligent optimization algorithms, the proposed algorithm effectively reduces resource allocation completion time and balances task execution time. As the number of tasks increases, the DSACO algorithm's allocation solution execution time is shorter than that of other intelligent algorithms, demonstrating its superior optimization capabilities in distributed cluster systems.

[0174] This paper combines fair allocation with combinatorial optimization. The proposed DSACO algorithm has a stronger ability to search for optimal solutions, effectively shortening the allocation time for batch tasks and conserving system resources. Compared with existing methods, this paper optimizes allocation schemes from multiple dimensions, including fairness, resource utilization, and allocation time. This improves the rapid allocation of resources and enhances the scalability, availability, and reliability of the system.

[0175] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only illustrative of the principles of the present invention. Without departing from the spirit and scope of the present invention, any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A distributed cluster resource autonomous scheduling method based on DSACO, characterized in that: The method comprises the following steps: Step 1: Building a system information repository RES based on node configuration information, node dynamic information, and network information, while simultaneously monitoring load status information on each node in the distributed cluster and link information between nodes in real time, and storing the information in the system information repository RES; Step 2: Evaluate the server nodes from the aspects of computing capacity, load, and resource utilization, calculate the comprehensive load and resource utilization of each node, and save them in the system information repository RES; Step 3: Based on the heterogeneity of nodes and user tasks in the server cluster, the resource scheduling problem is modeled as a multi-objective optimization problem, and a distributed resource autonomous allocation model is constructed; Step 4: Based on the distributed resource autonomous allocation model, and taking into account the differences in priority and resource requirements of user tasks, a fair allocation algorithm with priority iteration is used to construct a set of allocable tasks from all tasks within the available amount of cluster resources. Step 5: Based on the set of allocatable tasks and cluster node status information, a cluster resource decentralized search and competition optimization algorithm (DSACO) is constructed, so that each task searches for a server node with high adaptability based on its own resource demand characteristics, and determines a preliminary resource allocation plan. Step 6: Analyze the expected load changes of nodes, execute a competitive optimization algorithm on nodes with expected load differences greater than a preset standard, and determine a resource allocation plan. The competitive optimization algorithm includes: using a consistent hashing algorithm to randomly assign low-priority tasks to low-load nodes, and then performing a combinatorial optimization of the random assignment results based on a swarm intelligence optimization algorithm. In step 7, the expected load after resource allocation is evaluated using the comprehensive load and node pre-assigned task characteristics. If the expected load of the node does not exceed the set threshold, resource allocation is performed; otherwise, considering the fairness and priority of resource allocation, an elimination strategy is used for further processing to reduce the number of tasks to be executed to reduce the expected load.

2. The distributed cluster resource autonomous scheduling method based on DSACO according to claim 1, characterized in that: Step 1 builds the system information repository RES based on the node configuration information, node dynamic information and network information, specifically: Among them, RES static The static resources of the cluster include the number of CPU cores (CN), the node's memory size (MS), the disk size (DS), the clock frequency (CF), the disk IO rate (DR), and the network bandwidth (NB): RES dynamic It is the real-time status information of the node, which needs to be collected again each time resource allocation is performed, including CPU usage CN use , memory usage MS use , Disk Usage DS use and node bandwidth usage NB use RES network Dynamic information of the network, including average transmission delay NDE avg , the average throughput NT of the cluster where the node is located avg and the average packet loss rate NL avg .

3. The distributed cluster resource autonomous scheduling method based on DSACO according to claim 2, characterized in that: In step 2, the comprehensive load and resource utilization of each node are calculated based on the collected node load status information and computing power; The calculation formula for the comprehensive load is: Where NWL i is the comprehensive load of the i-th node. According to the characteristics of heterogeneous cluster tasks, the load is divided into IO load NL i and calculate the load CL i Two parts; The formula for calculating resource utilization is: Where, SRU i represents the resource utilization of the i-th server node, ω j represents the weight of resource j, SRC i ={SRC i1 , SRC i2 ...SRC im } represents the resource capacity vector of the i-th server node, where m is the number of resource types, SRC ij Indicates the total amount of resource j in the i-th node of the server, SRL i ={SRL i1 , SRL i2 …SRL im } represents the available capacity of resources in server i, SRL ij represents the available capacity of resource j in server i.

4. The distributed cluster resource autonomous scheduling method based on DSACO according to claim 3 is characterized in that: Step 3 specifically includes: Step 3.1, establish cluster system model MODEL system : MODEL system =(S,R,U,W) Among them, S represents the cluster node set, R corresponds to the node resource set, U represents the system user set, and W represents the user's task set; Step 3.2: Establish resource scheduling optimization objectives and constraints: (1) Fairness in resource allocation: Among them, f fairness The fairness of resource allocation is evaluated by the variance of the maximum resource allocation amount of the user, UTR u represents the maximum total amount of resource allocation for user u, Indicates the average of the total amount of resource allocation, n represents the number of users in the cluster, URV u ={URV u1 , URV u2 ,...,URV um } represents the resource demand vector of user u, where URV uj represents the demand of user u for resource j, UTN u Indicates the number of tasks of user u that has obtained resources, RL={RL1,RL2…RL m } represents the cluster resource availability vector, RL j represents the available amount of cluster resource j, and U is the user set; Resource allocation should ensure that the total amount of allocated resources is less than the available amount of cluster resources. The judgment method is: (2) Cluster load balancing: Among them, f loadbalance For cluster load balancing, represents the average load of the cluster, sn represents the number of server nodes, and is evaluated by the variance of the total load of each node; (3) Cluster resource utilization: Among them, f usage is the cluster resource utilization; (4) Resource scheduling time: The resource scheduling time f is determined by monitoring the time interval from the start of each task scheduling to the determination of the mapping relationship and submission to the node. time The maximum time interval of the currently scheduled task is calculated as follows: f time =max(w end -w start )w∈W Where w start represents the start time of resource scheduling, i.e. the start time of task fair selection, w end Indicates the time when the mapping relationship is finally determined; Step 3.3: Use the linear regression model to solve the multi-objective optimization problem of the resource scheduling process and establish a distributed resource autonomous allocation model: Where, ξ f ,ξ t ,ξ u ,ξ l Corresponding weights for different optimization objectives; The model optimization goal is to minimize f.

5. The distributed cluster resource autonomous scheduling method based on DSACO according to claim 4 is characterized in that: The fair allocation algorithm of priority iteration in step 4 is specifically as follows: when the total task resource demand is greater than the available cluster resources and all tasks cannot be allocated, a set of tasks that can currently be allocated is selected from all tasks.

6. The distributed cluster resource autonomous scheduling method based on DSACO according to claim 5, characterized in that: Step 4 specifically includes: Step 4.1: Considering the user's task priority and the amount of resources allocated to the user, the minimum resource allocator is obtained by the following formula: Among them, priority u Indicates task priority; represents the dominant resource of user u, Indicates that user u has demand; The calculation formula is: Step 4.2, allocating resources for the tasks of the minimum resource allocator; If there are multiple minimum resource allocators at the same time, the priority is determined and the minimum resource allocator with the highest priority is selected for allocation; Step 4.3: Loop through steps 4.1 and 4.2 until the total resource requirements of the selected tasks reach the current available cluster resources, thereby determining the allocatable task set WSL. Step 4.4, increase the priority of the currently unselected tasks outside the assignable task set WSL.

7. The distributed cluster resource autonomous scheduling method based on DSACO according to claim 6, characterized in that: Step 5 specifically includes: Phase 1: Determine decision variables and construct search space: Step 5.1: Construct an algorithm search space based on the server resources and task characteristics. This includes: Step 5.1.1: Analyze the allocatable task set WSL and determine the number of dimensions of the search space to be the number of resource requirement types in the task requirement set; Step 5.1.2: Analyze the available resources of the cluster nodes and determine the boundary value of each dimension as the maximum available amount of cluster node resources; Step 5.1.3, locate the node to the corresponding position in the search space according to the server node resource availability vector; Step 5.2: Determine the initial location of the task based on the task resource demand vector and the available resources of the cluster nodes; Step 5.3: Generate the initial direction and speed of each task. The speed of task w is v and the direction is o. The total number of iterations of the search process is δ. Phase 2: Determine the allocation results through decentralized search, including: Step 5.4: Start iterative search, calculate the fitness value of the task at the current node according to the following formula, and update the historical optimal fitness value of the task; FIT i =a T RUT i +a D RUD i +a L NWL i Where FIT i Indicates the fitness value of the current task and the i-th node, RUT i represents the change in resource utilization after the i-th node allocates resources to the task w of the current user u; α T , α D and α L Represents the weight parameters of the three indicators respectively; RUT i Calculation formula: Where, URV wj represents the demand of task w for resource j; RUD i Represents the deviation of the current server node resource utilization from the global average resource utilization, RUD i The calculation formula is: Where, SRU i’ represents the resource utilization of the i'th node of the server; Step 5.5, determine the moving direction of the next task in the search space; The optimal search direction is determined based on the task resource demand vector, and a random interference vector is introduced for interference. The search direction and speed of task w of user u at a certain moment are shown as follows: in, is the weight of the random vector, which is used to adjust the optimization ability and convergence speed. Random is the random interference vector, which consists of m random numbers κ j Composition, O is the vector of speed and direction; In step 5.6, the above iterative process is repeated until the final number of iterations is reached, and the fitness value of the current position is compared with the historical optimal position. The task is moved to the node with higher fitness, the allocation result is confirmed, and the preliminary resource allocation is completed.

8. The distributed cluster resource autonomous scheduling method based on DSACO according to claim 7, characterized in that: In step 6, the node expected load change analysis is performed, and the competitive optimization algorithm is executed on the nodes with large expected load differences, including: If multiple tasks are eventually optimized for the same node, resources are directly allocated if the node has sufficient resources; otherwise, a competitive optimization algorithm is executed.

9. The distributed cluster resource autonomous scheduling method based on DSACO according to claim 8, characterized in that: Step 6 specifically includes: Step 6.1: For each server node with task conflicts, select the high-priority task for allocation based on task priority, and then integrate the unselected tasks into the task set to be optimized WCL. Task conflicts refer to tasks where the resource requirements are greater than the available resources of the node. In step 6.2, the consistent hashing algorithm is used for random allocation. All nodes are connected into a hash ring, and tasks in the WCL are randomly assigned to nodes. If a node has insufficient resources, it moves to the next node in the specified direction. After a certain number of moves, if there are still tasks that cannot find a corresponding node, the task is removed and the priority is iterated to participate in the next round of resource allocation. Step 6.3: Optimize the allocation results using the Black Widow Optimization Algorithm to obtain an optimized resource allocation plan. This includes: In step 6.3.1, each server is abstracted as an individual in the Black Widow population, and the optimization rate pr, elimination rate cr, optimization times on, optimization threshold ot, and the number of servers participating in the optimization N are defined in the algorithm. The cluster nodes are sorted according to the resource utilization rate, and N×pr individuals with low resource utilization in the cluster are selected for optimization based on the optimization rate pr. The individuals participating in the optimization are randomly grouped into teams for reproduction. The reproduction process is shown in the following formula: Among them, S1 and S2 are the node pairs to be optimized, S ` 1 and S ` 2 is the optimized node pair, α is a randomly generated probability array of auxiliary reproduction; Step 6.3.2: Calculate the fitness of the newly bred individuals, reorder the N individuals, select N × cr individuals with high fitness in the population based on the elimination rate cr, and then optimize the remaining individuals; In step 6.3.3, if the upper limit of the number of combined optimizations is reached or the optimization effect cannot reach the optimization threshold, the algorithm terminates and the optimized resource allocation plan is obtained.

10. The distributed cluster resource autonomous scheduling method based on DSACO according to claim 9, characterized in that: Step 7 specifically includes: For the resource allocation scheme obtained in step 6, calculate the expected comprehensive load and resource utilization of each node, and compare whether the expected result exceeds the node load threshold. If NWL i If the load threshold is exceeded, some tasks will be eliminated to ensure the normal operation of the cluster; Among them, the elimination strategy is as follows: That is, the priority of the task set on the slave node w , resource allocation fairness f fairness The fitness value FIT of the task at the node is comprehensively evaluated, and the task with the smallest evaluation result is eliminated.

Images