Distributed cluster resource autonomous scheduling method based on DSACO

Abstract

The invention discloses a distributed cluster resource autonomous scheduling method based on DSACO, and the method mainly comprises the steps: monitoring the state of each node and network topology information in a cluster system in real time, and carrying out the dynamic tracking of the operation condition of a distributed cluster; secondly, constructing a distributed resource autonomous scheduling model based on a multi-objective planning model and a swarm intelligence optimization thought; building a multi-stage resource allocation algorithm (DSACO) based on an autonomous resource allocation model, fairly selecting a task set, carrying out dispersed search on different tasks, carrying out further optimization processing on a preliminary allocation result, and determining a better resource allocation scheme; and finally, performing judgment and processing according to the QoS evaluation index, and enhancing the cluster reliability under a high-load condition. According to the method, resource allocation is optimized from multiple dimensions of resource allocation fairness, resource utilization rate and comprehensive load, the rapid batch allocation capability of distributed cluster resources in a large-scale system can be improved, and the service quality under the high-load condition is improved.

Description

Technical Field

[0001] The invention belongs to the technical field of high reliability and high availability, and in particular to a distributed cluster resource autonomous scheduling method based on DSACO. Background Art

[0002] In recent years, with the vigorous development of the Internet, the application of distributed computing has become more and more extensive, and has been widely used in various industries such as finance, medical care, e-commerce, social media, and the Internet of Things. Distributed clusters can effectively improve data processing capabilities, system reliability, and user experience by distributing computing tasks and data storage on multiple nodes. For example, when building a large website, distributed computing can distribute user requests to multiple servers through load balancing technology, thereby significantly improving the response speed and stability of the system. In addition, distributed clusters also play a key role in fields such as big data processing, cloud computing, and edge computing, supporting complex data analysis, real-time data processing, and a wide range of service deployment.

[0003] However, most existing research focuses on aspects such as data consistency, load balancing, and fault tolerance mechanisms, while ignoring the optimization of resource allocation in distributed environments. Resource allocation is crucial in distributed clusters because it directly affects the performance, efficiency, and cost of the system. In the case of improper resource allocation, the system may suffer from problems such as resource waste, task congestion, and response delays, thereby reducing the overall system operation efficiency and user experience. In particular, in scenarios where resource requirements change dynamically, such as high concurrent access and real-time data processing, the challenge of resource allocation is more prominent.

[0004] For example, cloud service providers dynamically provide users with virtual machine services. This process is achieved by deploying virtual machine monitors on physical machines. Placing different types of virtual machines on a single physical machine requires dividing the physical machine's resources to different degrees. If the resource allocation of thousands of virtual machines cannot optimally divide the physical machine resources, then the data center will need more physical machines to meet the needs of cloud customers. Adding more physical machines to the data center will lead to higher energy consumption, hardware costs, and maintenance costs, while causing serious resource waste. Therefore, it is crucial to study fair and effective resource allocation strategies. In large-scale cloud computing systems, effective and fair resource allocation is particularly challenging. This is because: (1) there are many types of resources, such as computing power, storage space, and network bandwidth; (2) workloads have diverse and rapidly changing demands for these resources; (3) server resource capacity is heterogeneous, and the performance and configuration of different servers vary; (4) workloads may have other restrictions on the use of servers, such as geographical location, network latency, and other factors. These challenges make it very complicated to achieve efficient and fair resource allocation in practical applications.

[0005] Researchers have proposed a variety of optimization algorithms and strategies for different types of distributed clusters and application scenarios. The main research directions are divided into traditional mathematical optimization models, swarm intelligence optimization strategies, and resource allocation strategies combined with machine learning. These methods can improve resource scheduling and the balance of task allocation to a certain extent, and at the same time achieve more comprehensive resource management and monitoring. Typical work such as: M. Wang et al. proposed a Hadoop resource scheduling algorithm TMSA based on task allocation. By calculating and analyzing the degree of adaptation of each task to each node, the prediction algorithm is used to predict the final completion time of the task. By comparing the prediction results, the final allocation plan is obtained, thereby optimizing the overall completion time of the task. However, with the increase in the number of tasks, the amount of algorithm calculation increases significantly, and the performance is unstable; HU et al. analyzed the real-time performance of nodes and their configuration, and proposed a cluster resource allocation algorithm with adaptive node real-time performance. By analyzing the dynamic and static resource weights, the selection is combined with the task type to achieve the adaptation of node real-time performance to the task type. The node with the best real-time performance is assigned to the task, thereby improving the overall performance of the cluster system. However, the weights of resources in this paper are fixed, so it is not suitable for clusters with higher complexity. Mirzaei et al. proposed a new architecture for offloading jobs and allocating resources in the Internet of Things. The MFO algorithm is combined with the optimization method of adversarial learning (OBL) to allocate resources to avoid system load imbalance and ensure data accuracy. However, due to the high sensitivity of algorithm parameters, performance may be unstable in complex cluster scenarios. Muhammad et al. proposed a method of combining a load balancing task scheduler with a device predictor based on machine learning. The predictor is used to predict the execution time on CPU and GPU devices. Devices with shorter predicted execution time are considered to be suitable for this specific task. A task scheduler based on work stealing is designed as part of the solution to optimize the prediction results. However, due to the high node resource consumption of machine learning and the possibility of conflicts, the second-stage task scheduling becomes complicated and is not applicable to clusters with large task volumes and complex businesses.

[0006] On the other hand, in order to deal with the problem of resource allocation fairness in cloud computing environments, DRF (Dominant Resource Fairness) was proposed and promoted as a fair allocation strategy. DRF aims to balance the global dominant share of each user in multi-resource application scenarios. This definition is based on the highest proportion of any resource allocated to the user in the resource pool. DRF not only significantly improves the fairness of resource allocation, but its extended heterogeneous DRF (Heterogeneous Dominant Resource Fairness, HDRF) is also introduced as an advanced multi-resource scheduling algorithm to effectively prevent starvation caused by uneven resource allocation of jobs in different data centers. However, there is still room for further improvement in the optimization of resource allocation rate and the refinement of cluster load management.

[0007] In summary, although good research results have been achieved in the scheduling of distributed cluster resources, the following issues still need further study: (1) Insufficient balance between multi-dimensional performance indicators such as resource utilization, fairness, and cluster load; (2) Limited adaptability to complex dynamic environments; (3) Algorithm execution efficiency and resource usage need to be optimized. Summary of the invention

[0008] The purpose of the present invention is to provide a distributed cluster resource autonomous scheduling method based on DSACO to address the problems existing in the above-mentioned prior art. The present invention focuses on optimizing the balance of multi-dimensional performance indicators, improving adaptability to complex dynamic environments, and optimizing algorithm execution efficiency and resource occupancy, in order to contribute new solutions and ideas to this field.

[0009] The technical solution to achieve the purpose of the present invention is: on the one hand, a distributed cluster resource autonomous scheduling method based on DSACO is provided, the method comprising:

[0010] Step 1, building a system information repository RES based on node configuration information, node dynamic information and network information, and monitoring the load status information on each node in the distributed cluster and the link information between nodes in real time, and storing them in the system information repository RES;

[0011] Step 2, evaluating the server nodes from the aspects of computing capacity, load condition, and resource utilization, calculating the comprehensive load and resource utilization of each node, and storing them in the system information repository RES;

[0012] Step 3: Based on the heterogeneity of nodes and user tasks in the server cluster, the resource scheduling problem is modeled as a multi-objective optimization problem, and a distributed resource autonomous allocation model is constructed;

[0013] Step 4: Based on the distributed resource autonomous allocation model, according to the differences in priorities and resource requirements of user tasks, a fair allocation algorithm with priority iteration is used to construct a set of allocatable tasks from all tasks within the limit of available cluster resources;

[0014] Step 5: Based on the allocatable task set and cluster node status information, a cluster resource decentralized search and competition optimization algorithm DSACO is constructed, so that each task searches for a server node with high fitness according to its own resource demand characteristics, and determines a preliminary resource allocation plan;

[0015] Step 6, analyzing the expected load change of nodes, executing a competitive optimization algorithm for nodes with expected load differences greater than a preset standard, and determining a resource allocation scheme; the competitive optimization algorithm includes: using a consistent hashing algorithm to first randomly assign low-priority tasks to low-load nodes, and further performing combinatorial optimization on the random assignment results based on a swarm intelligence optimization algorithm;

[0016] Step 7: Use the comprehensive load and node pre-allocated task characteristics to evaluate the expected load after resource allocation. If the expected load of the node does not exceed the set threshold, resource allocation is performed; otherwise, considering the fairness and priority of resource allocation, an elimination strategy is used for further processing to reduce the number of tasks to be executed to reduce the expected load.

[0017] On the other hand, a distributed cluster resource autonomous scheduling system based on DSACO is provided, the system comprising:

[0018] The first module is used to build a system information base RES based on node configuration information, node dynamic information and network information, and monitor the load status information on each node in the distributed cluster and the link information between nodes in real time, and save them in the system information base RES;

[0019] The second module is used to evaluate the server nodes from the aspects of computing capacity, load condition and resource utilization, calculate the comprehensive load and resource utilization of each node, and save them in the system information base RES;

[0020] The third module is used to model the resource scheduling problem as a multi-objective optimization problem according to the heterogeneity of nodes and user tasks in the server cluster, and to build a distributed resource autonomous allocation model;

[0021] The fourth module is used to construct an allocatable task set from all tasks within the available amount of cluster resources based on the distributed resource autonomous allocation model and according to the differences in priorities and resource requirements of user tasks through a fair allocation algorithm with priority iteration;

[0022] The fifth module is used to construct a cluster resource decentralized search and competition optimization algorithm DSACO based on the set of allocatable tasks and cluster node status information, so that each task searches for a server node with high fitness according to its own resource demand characteristics and determines a preliminary resource allocation plan;

[0023] The sixth module is used to analyze the expected load change of the node, execute the competitive optimization algorithm for the nodes with expected load differences greater than the preset standard, and determine the resource allocation plan; the competitive optimization algorithm includes: using the consistent hashing algorithm to randomly assign low-priority tasks to low-load nodes, and further combining and optimizing the random allocation results based on the swarm intelligence optimization algorithm;

[0024] The seventh module is used to evaluate the expected load after resource allocation using the comprehensive load and node pre-allocated task characteristics. If the expected load of the node does not exceed the set threshold, resource allocation is performed; otherwise, considering the fairness and priority of resource allocation, an elimination strategy is used for further processing to reduce the number of tasks to be executed to reduce the expected load.

[0025] Compared with the prior art, the present invention has the following significant advantages:

[0026] (1) It takes into account the fairness of task allocation, resource utilization, and load, thus improving the overall performance and user experience of the cluster system.

[0027] (2) A priority-iterative dominant resource fair allocation method is proposed, which comprehensively considers three aspects: task priority, task submission time, and the number of allocated resources to ensure the fairness of resource allocation.

[0028] (3) A resource autonomous scheduling model is constructed to solve the resource allocation problem. A method for constructing a resource search space is proposed, which can comprehensively analyze the distribution characteristics of cluster resources in real time, making the resource allocation plan more complete and accurate.

[0029] (4) Based on the characteristics of resources in heterogeneous clusters, a multi-stage optimization algorithm, the decentralized search and competition optimization algorithm (DSACO), is proposed. This intelligent optimization algorithm has a stronger ability to search for the optimal solution, solves the resource competition problem of the local optimal solution, and shortens the processing time of searching for the optimal resource allocation solution.

[0030] The present invention is further described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a structural diagram of the distributed cluster resource autonomous scheduling method based on DSACO.

[0032] Figure 2 Flowchart of the distributed cluster resource autonomous scheduling method based on DSACO.

[0033] Figure 3 A comparison chart of cluster node resource utilization experimental results in one embodiment.

[0034] Figure 4 The figure is a comparison chart of task completion time in one embodiment. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0036] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0037] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0038] In traditional cluster service systems, resource allocation algorithms pay little attention to resource utilization and have low allocation efficiency. Figure 1 and Figure 2 The present invention provides a distributed cluster resource autonomous scheduling method based on DSACO, the method comprising the following steps:

[0039] Step 1, building a system information repository RES based on node configuration information, node dynamic information and network information, and monitoring the load status information on each node in the distributed cluster and the link information between nodes in real time, and storing them in the system information repository RES;

[0040] Step 2, evaluating the server nodes from the aspects of computing capacity, load condition, and resource utilization, calculating the comprehensive load and resource utilization of each node, and storing them in the system information repository RES;

[0041] Step 3: Based on the heterogeneity of nodes and user tasks in the server cluster, the resource scheduling problem is modeled as a multi-objective optimization problem, and a distributed resource autonomous allocation model is constructed;

[0042] Step 4: Based on the distributed resource autonomous allocation model, according to the differences in priorities and resource requirements of user tasks, a fair allocation algorithm with priority iteration is used to construct a set of allocatable tasks from all tasks within the limit of available cluster resources;

[0043] Step 5: Based on the allocatable task set and cluster node status information, a cluster resource decentralized search and competitive optimization algorithm DSACO (Decentralized Search and Competitive Optimization) is constructed, so that each task searches for a server node with high fitness according to its own resource demand characteristics and determines a preliminary resource allocation plan;

[0044] Step 6, analyzing the expected load change of nodes, executing a competitive optimization algorithm for nodes with expected load differences greater than a preset standard, and determining a resource allocation scheme; the competitive optimization algorithm includes: using a consistent hashing algorithm to first randomly assign low-priority tasks to low-load nodes, and further performing combinatorial optimization on the random assignment results based on a swarm intelligence optimization algorithm;

[0045] Step 7: Use the comprehensive load and node pre-allocated task characteristics to evaluate the expected load after resource allocation. If the expected load of the node does not exceed the set threshold, resource allocation is performed; otherwise, considering the fairness and priority of resource allocation, an elimination strategy is used for further processing to reduce the number of tasks to be executed to reduce the expected load.

[0046] Furthermore, in one of the embodiments, the step 1 of constructing a system information repository RES based on the node configuration information, the node dynamic information and the network information is specifically as follows:

[0047] RES=(RES static ,RES dynamic ,RES network #(1)

[0048] Among them, RES static It is the static resource of the cluster, that is, the basic configuration information of the server node and the network. When a new service node joins the cluster, it collects resource information, which remains unchanged during the subsequent system operation. The detailed description of the static resource is as follows:

[0049] RES static =(CN,MS,DS,CF,DR,NB)#(2)

[0050] Where CN is the number of CPU cores of the server node, MS is the memory size of the node, the disk size is DS, CF is the clock frequency, DR is the disk IO rate, and NB is the network bandwidth;

[0051] Among them, RES dynamic is the real-time status information of the node, which is collected again each time resource allocation is performed to monitor the status of cluster nodes in real time. The detailed description of the resources is as follows:

[0052] RES dynamic =(CN use ,MS use ,DS use ,NB use )#(3)

[0053] In the formula, CN use Indicates CPU usage, MS use Indicates memory usage, NB use is the node bandwidth utilization, DS use For disk usage;

[0054] Among them, RES networ k is the network state parameter of the node, which is described in detail in formula (4):

[0055] RES network =(NDE avg ,NT avg ,NL avg )#(4)

[0056] Where NT avg is the average throughput of the cluster where the node is located, NL avg is the average packet loss rate of the cluster where the node is located, NDE avg is the average transmission delay.

[0057] Further, in one of the embodiments, in step 2, the comprehensive load and resource utilization of each node are calculated based on the collected node load status information and computing capacity;

[0058] The calculation formula of the comprehensive load is as follows:

[0059]

[0060] Where NWL i is the comprehensive load of the i-th node. According to the characteristics of heterogeneous cluster tasks, the load is divided into IO load NL i and calculate the load CL iThere are two parts. The network IO load is caused by complex computing-related tasks and is mainly determined by performance parameters such as network bandwidth, network latency, and network transmission rate. The computing load is caused by big data computing-related tasks and is determined by the number of CPU cores, memory size, clock frequency, and disk IO rate.

[0061] The calculation formula of resource utilization is shown in formula (6):

[0062]

[0063] In the formula, SRU i represents the resource utilization of the i-th server node, ω j represents the weight of resource j, SRC i ={SRC i1 ,SRC i2 …SRC im} represents the resource capacity vector of the i-th server node, where m is the number of resource types, SRC ij represents the total amount of resource j in the i-th node of the server, SRL i ={SRL i1 ,SRL i2 …SRL im} represents the available capacity of resources in server i, SRL ij represents the available capacity of resource j in server i.

[0064] Further, in one embodiment, step 3 specifically includes:

[0065] Step 3.1: Establish cluster system model MODEL system :

[0066] MODEL system =(S,R,U,W)#(7)

[0067] Among them, S represents the cluster node set, R corresponds to the node resource set, U represents the system user set, and W represents the user's task set;

[0068] Step 3.2, establish resource scheduling optimization objectives and constraints:

[0069] The problem to be solved in resource allocation is to reasonably allocate cluster resources to user tasks and realize resource scheduling, which is essentially to establish a mapping relationship from tasks to cluster nodes. To this end, the present invention proposes the following optimization goals: (1) fairness of task allocation; (2) resource utilization; (3) load balancing; and (4) algorithm execution time.

[0070] (1) Resource allocation fairness, which is used to determine the balance of resource allocation to users and improve user experience. The calculation formula is:

[0071]

[0072] Among them, f fairness The fairness of resource allocation is evaluated by the variance of the maximum resource allocation quantity of the user, UTR u represents the maximum total amount of resource allocation for user u, represents the average of the total amount of resource allocation, n represents the number of users in the cluster, URV u ={URV u1 ,URV u2 ,…,URV um} represents the resource demand vector of user u, where URV uj represents the demand of user u for resource j, UTN u represents the number of tasks of user u that has obtained resources, RL={RL1,RL2…RL m} represents the cluster resource availability vector, RL j represents the available amount of cluster resource j, and U is the user set;

[0073] Resource allocation should satisfy the requirement that the total amount of allocated resources is less than the available amount of cluster resources. The judgment method is:

[0074]

[0075] (2) Cluster load balancing, which is used to evaluate the balance of load balancing and resource utilization. The calculation formula is:

[0076]

[0077] Among them, f loadbalance For cluster load balancing, represents the average load of the cluster, sn represents the number of server nodes, and is evaluated by the variance of the total load of each node;

[0078] (3) Cluster resource utilization, which is used to evaluate the rationality of resource allocation and further compensate for the problem of ineffective resource utilization caused by fairness. The calculation formula is:

[0079]

[0080] Among them, f usage is the cluster resource utilization;

[0081] (4) Resource scheduling time: used to evaluate the efficiency of algorithm execution. By monitoring the time interval from the start of scheduling each task to determining the mapping relationship and submitting it to the node, the resource scheduling time f is determined. time It is the maximum time interval of the currently scheduled task, and the calculation formula is:

[0082] f time =max(w end -w start )w∈W#(12)

[0083] In the formula, w start represents the start time of resource scheduling, i.e. the start time of task fair selection, w end Indicates the time when the mapping relationship was finally determined;

[0084] Step 3.3, use the linear regression model to solve the multi-objective optimization problem of the resource scheduling process and establish a distributed resource autonomous allocation model:

[0085]

[0086] In the formula, ξ f ,ξ t ,ξ u ,ξ l Corresponding weights for different optimization objectives;

[0087] In order to unify the optimization goal, the inverse of the resource utilization is taken. Therefore, the smaller the value of each optimization goal is, the better the allocation plan given by the resource scheduling method is. The overall model optimization goal is to minimize f.

[0088] Furthermore, in one of the embodiments, the priority iterative fair allocation algorithm in step 4 is specifically as follows: when the total task resource demand is greater than the available cluster resource and all tasks cannot be allocated, a set of tasks that can currently be allocated is selected from all tasks.

[0089] Here, a fair allocation algorithm with priority iteration is designed to analyze the balance problem between user priority and fair allocation.

[0090] If only task priority is considered, it will cause low resource utilization. If only fairness is analyzed, some tasks may not be executed for a long time. Therefore, the present invention designs a priority-iterative task fair allocation algorithm. When the total task resource demand is greater than the available cluster resources and all tasks cannot be allocated, a set of tasks that can currently be allocated is selected from all tasks to ensure the fairness of resource allocation.

[0091] Further, in one embodiment, step 4 specifically includes:

[0092] Step 4.1, comprehensively consider the user's task priority and the number of resources allocated to the user, and calculate the minimum resource allocator through the following formula;

[0093]

[0094] Among them, UTN={UTN1,UTN2…UTN n}, UTN u Indicates the number of tasks to which resources have been allocated for user u, UTN u Satisfy:∑ u＝1..n UTN u *URV uj ≤RL j , That is, the number of allocated resources is less than the available amount of cluster resources; URV u ={URV u1 ,URV u2 …URV um} represents the task resource requirement vector of user u, RL={RL1,RL2…RL m} represents the set of available resources in the cluster, priority u Indicates the task priority; represents the user's dominant resource, Indicates that user u has Quantity of demand; The calculation formula is:

[0095]

[0096] Step 4.2, allocating resources for the tasks of the minimum resource allocator;

[0097] Here, if there are multiple minimum resource allocators at the same time, the priority is determined and the minimum resource allocator with the highest priority is selected for allocation;

[0098] Step 4.3, loop through steps 4.1 to 4.2 until the total resource requirements of the selected tasks reach the current available cluster resources (the available resources are insufficient when the next task is selected), stop selecting, and thereby determine the allocatable task set WSL;

[0099]

[0100] Step 4.4, increase the priority of currently unselected tasks outside the allocatable task set WSL to ensure that these tasks are selected first next time.

[0101] Further, in one embodiment, step 5 specifically includes:

[0102] Phase 1: Determine decision variables and construct search space:

[0103] Step 5.1: construct an algorithm search space based on the characteristics of server resources and tasks; specifically, it includes:

[0104] Step 5.1.1, analyze the allocatable task set WSL and determine the number of dimensions of the search space to be the number of types of resource requirements in the task requirement set;

[0105] Step 5.1.2, analyze the available resources of the cluster nodes and determine the boundary value of each dimension as the maximum available amount of cluster node resources;

[0106] Step 5.1.3, according to the server node resource availability vector, locate the node to the corresponding position in the search space;

[0107] Step 5.2, determine the initial position of the task according to the task resource demand vector and the available resources of the cluster node; the steps are as follows:

[0108] Step 5.2.1, calculate and mark the minimum available resources of each node;

[0109] Step 5.2.2, analyze the dominant resource of the task and map the task to the node with the smallest available dominant resource;

[0110] Step 5.3, generate the initial direction and speed of each task, the speed of task w is v, the direction is o, and the total number of iterations of the search process is δ;

[0111] Phase 2: The allocation results are obtained through decentralized search, including:

[0112] Step 5.4, start iterative search, calculate the fitness value of the task at the current node according to the following formula, and update the historical optimal fitness value of the task;

[0113] FIT i =α T RUT i +α D RUD i +α L NWL i #(17)

[0114] Where FIT i Represents the fitness value of the current task and the i-th node, RUT i Represents the resource utilization change after the i-th node allocates resources to the task w of the current user u; NWL i is the comprehensive load of node i after the task is assigned to the node, and the load is defined as shown in formula (5); α T , α D and α L Respectively represent the weight parameters of the three indicators; RUT i Calculation formula:

[0115]

[0116] RUD i Represents the deviation of the current server node resource utilization from the global average resource utilization, RUD i The calculation formula is:

[0117]

[0118] Among them, SRU i` Indicates the resource utilization of the i-th node of the server;

[0119] Step 5.5, determine the moving direction of the next task in the search space;

[0120] The optimal search direction is determined according to the task resource demand vector, and a random interference vector is introduced to interfere to prevent the final search results from being too concentrated, so as to improve the global optimization capability. The search direction and speed of task w of user u at a certain moment are shown in the following formula:

[0121]

[0122] in, is the weight of the random vector, which is used to adjust the optimization ability and convergence speed, Random is the random interference vector, and O is the vector of speed and direction;

[0123] Step 5.6, loop the above iterative process until the final number of iterations is reached, and compare the fitness value of the current position with the historical optimal position, move the task to the node with high fitness, confirm the allocation result, and complete the preliminary resource allocation.

[0124] Further, in one of the embodiments, the analysis of the expected load change of the nodes in step 6 and the execution of the competition optimization algorithm for the nodes with large expected load differences specifically include:

[0125] When multiple tasks eventually find the same node, if the node has sufficient resources, resources are directly allocated; otherwise, the competitive optimization algorithm is executed.

[0126] In some embodiments here, step 6 specifically includes:

[0127] Step 6.1, for each server node with task conflicts, select high-priority tasks for allocation based on task priority, and then integrate unselected tasks into a task set to be optimized WCL; the task conflict refers to the task resource demand being greater than the node available resource;

[0128] Step 6.2: Use the consistent hashing algorithm for random allocation, connect all nodes into a hash ring, and randomly allocate tasks in the WCL to nodes. If the node resources are insufficient, move to the next node in the specified direction. After a certain number of moves, if there are still tasks that cannot find the corresponding nodes, remove the tasks and perform priority iteration to participate in the next round of resource allocation.

[0129] Step 6.3, optimize the random allocation result of step 6.2, specifically combine the idea of ​​the black widow optimization algorithm, exchange the tasks of the two machines to compare whether a better fitness value can be obtained, if the fitness is better, determine the exchange; if it cannot achieve a better effect, eliminate it, and get a further allocation plan. The optimization steps are as follows:

[0130] Step 6.3.1, abstract each server as an individual in the black widow population, define the optimization rate pr, elimination rate cr, optimization times on, optimization threshold ot, and the number of servers participating in the optimization N in the algorithm;

[0131] The cluster nodes are sorted according to the resource utilization rate, and N×pr individuals with low resource utilization rate in the cluster are selected to participate in the optimization based on the optimization rate pr; the individuals participating in the optimization are randomly grouped for reproduction, and the reproduction process is shown in the following formula:

[0132]

[0133] Among them, S1 and S2 are the selected node pairs to be optimized, S`1 and S`2 are the optimized node pairs, and α is a randomly generated probability array of auxiliary reproduction;

[0134] Step 6.3.2, calculate the fitness value of the new individuals bred according to formula (17), re-rank the N individuals, select N×cr individuals with high fitness in the population based on the elimination rate cr, and optimize the remaining individuals;

[0135] Step 6.3.3: If the upper limit of the number of combined optimizations is reached or the optimization effect cannot reach the optimization threshold, the algorithm terminates and the optimized resource allocation plan is obtained.

[0136] Since the allocation result in step 6.3 may not completely meet expectations in actual operation, there may be a situation where the actual operation result exceeds the expected result and causes the server node to overload and crash, so further redundant fault tolerance is required for the result.

[0137] Further, in one embodiment, step 7 specifically includes:

[0138] For the resource allocation scheme obtained in step 6, evaluate it according to equations (5) and (6), calculate the expected comprehensive load and resource utilization of each node, and compare whether the expected result exceeds the node load threshold. If NWL i If the load threshold is exceeded, some tasks will be eliminated to ensure the normal operation of the cluster;

[0139] Among them, the elimination strategy is as follows:

[0140]

[0141] That is, the priority of the task set on the slave node w , resource allocation fairness f fairness The fitness value FIT of the task at the node is comprehensively evaluated, and the task with the smallest evaluation result is eliminated.

[0142] In one embodiment, a distributed cluster resource autonomous scheduling system based on DSACO is provided, the system comprising:

[0143] The first module is used to build a system information base RES based on node configuration information, node dynamic information and network information, and monitor the load status information on each node in the distributed cluster and the link information between nodes in real time, and save them in the system information base RES;

[0144] The second module is used to evaluate the server nodes from the aspects of computing capacity, load condition and resource utilization, calculate the comprehensive load and resource utilization of each node, and save them in the system information base RES;

[0145] The third module is used to model the resource scheduling problem as a multi-objective optimization problem according to the heterogeneity of nodes and user tasks in the server cluster, and to build a distributed resource autonomous allocation model;

[0146] The fourth module is used to construct an allocatable task set from all tasks within the available amount of cluster resources based on the distributed resource autonomous allocation model and according to the differences in priorities and resource requirements of user tasks through a fair allocation algorithm with priority iteration;

[0147] The fifth module is used to construct a cluster resource decentralized search and competition optimization algorithm DSACO based on the set of allocatable tasks and cluster node status information, so that each task searches for a server node with high fitness according to its own resource demand characteristics and determines a preliminary resource allocation plan;

[0148] The sixth module is used to analyze the expected load change of the node, execute the competitive optimization algorithm for the nodes with expected load differences greater than the preset standard, and determine the resource allocation plan; the competitive optimization algorithm includes: using the consistent hashing algorithm to randomly assign low-priority tasks to low-load nodes, and further combining and optimizing the random allocation results based on the swarm intelligence optimization algorithm;

[0149] The seventh module is used to evaluate the expected load after resource allocation using the comprehensive load and node pre-allocated task characteristics. If the expected load of the node does not exceed the set threshold, resource allocation is performed; otherwise, considering the fairness and priority of resource allocation, an elimination strategy is used for further processing to reduce the number of tasks to be executed to reduce the expected load.

[0150] For the specific limitations of the distributed cluster resource autonomous scheduling system based on DSACO, please refer to the limitations of the distributed cluster resource autonomous scheduling method based on DSACO above, which will not be repeated here. Each module in the above-mentioned distributed cluster resource autonomous scheduling method based on DSACO can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0151] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following is achieved:

[0152] Step 1, building a system information repository RES based on node configuration information, node dynamic information and network information, and monitoring the load status information on each node in the distributed cluster and the link information between nodes in real time, and storing them in the system information repository RES;

[0153] Step 2, evaluating the server nodes from the aspects of computing capacity, load condition, and resource utilization, calculating the comprehensive load and resource utilization of each node, and storing them in the system information repository RES;

[0154] Step 3: Based on the heterogeneity of nodes and user tasks in the server cluster, the resource scheduling problem is modeled as a multi-objective optimization problem, and a distributed resource autonomous allocation model is constructed;

[0155] Step 4: Based on the distributed resource autonomous allocation model, according to the differences in priorities and resource requirements of user tasks, a fair allocation algorithm with priority iteration is used to construct a set of allocatable tasks from all tasks within the limit of available cluster resources;

[0156] Step 5: Based on the allocatable task set and cluster node status information, a cluster resource decentralized search and competition optimization algorithm DSACO is constructed, so that each task searches for a server node with high fitness according to its own resource demand characteristics, and determines a preliminary resource allocation plan;

[0157] Step 6, analyzing the expected load change of nodes, executing a competitive optimization algorithm for nodes with expected load differences greater than a preset standard, and determining a resource allocation scheme; the competitive optimization algorithm includes: using a consistent hashing algorithm to first randomly assign low-priority tasks to low-load nodes, and further performing combinatorial optimization on the random assignment results based on a swarm intelligence optimization algorithm;

[0158] Step 7: Use the comprehensive load and node pre-allocated task characteristics to evaluate the expected load after resource allocation. If the expected load of the node does not exceed the set threshold, resource allocation is performed; otherwise, considering the fairness and priority of resource allocation, an elimination strategy is used for further processing to reduce the number of tasks to be executed to reduce the expected load.

[0159] For the specific limitations of each step, please refer to the limitations of the distributed cluster resource autonomous scheduling method based on DSACO in the above text, which will not be repeated here.

[0160] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the computer program implements:

[0161] Step 1, building a system information repository RES based on node configuration information, node dynamic information and network information, and monitoring the load status information on each node in the distributed cluster and the link information between nodes in real time, and storing them in the system information repository RES;

[0162] Step 2, evaluating the server nodes from the aspects of computing capacity, load condition, and resource utilization, calculating the comprehensive load and resource utilization of each node, and storing them in the system information repository RES;

[0163] Step 3: Based on the heterogeneity of nodes and user tasks in the server cluster, the resource scheduling problem is modeled as a multi-objective optimization problem, and a distributed resource autonomous allocation model is constructed;

[0164] Step 4: Based on the distributed resource autonomous allocation model, according to the differences in priorities and resource requirements of user tasks, a fair allocation algorithm with priority iteration is used to construct a set of allocatable tasks from all tasks within the limit of available cluster resources;

[0165] Step 5: Based on the allocatable task set and cluster node status information, a cluster resource decentralized search and competition optimization algorithm DSACO is constructed, so that each task searches for a server node with high fitness according to its own resource demand characteristics, and determines a preliminary resource allocation plan;

[0166] Step 6, analyzing the expected load change of nodes, executing a competitive optimization algorithm for nodes with expected load differences greater than a preset standard, and determining a resource allocation scheme; the competitive optimization algorithm includes: using a consistent hashing algorithm to first randomly assign low-priority tasks to low-load nodes, and further performing combinatorial optimization on the random assignment results based on a swarm intelligence optimization algorithm;

[0167] Step 7: Use the comprehensive load and node pre-allocated task characteristics to evaluate the expected load after resource allocation. If the expected load of the node does not exceed the set threshold, resource allocation is performed; otherwise, considering the fairness and priority of resource allocation, an elimination strategy is used for further processing to reduce the number of tasks to be executed to reduce the expected load.

[0168] For the specific limitations of each step, please refer to the limitations of the distributed cluster resource autonomous scheduling method based on DSACO in the above text, which will not be repeated here.

[0169] As a specific example, in one of the embodiments, the present invention is further verified and described in detail. A virtual machine cluster is built to compare the performance of the DSACO algorithm of the present invention with the Round Robin algorithm, the ATNPA algorithm, and the multi-objective particle swarm optimization algorithm (MOPSO). The MOPSO algorithm adopts a decomposition method, assigns an optimization sub-problem to each particle, and adopts a novel speed update strategy to speed up the convergence speed. The ATNPA algorithm is a resource allocation algorithm proposed by Hu et al. that considers dynamic and static factors, and can optimize cluster resource utilization and load balancing.

[0170] The distributed heterogeneous cluster system set up in the experiment includes 1 central node and 10 nodes with 4 different resource quantities. In the distributed cluster system, the central node monitors the status of other nodes in the cluster, analyzes user task requirements, and allocates resources. This example does not consider the task parsing steps. This example designs 3 types of tasks: IO-intensive, CPU-intensive, and mixed-intensive tasks based on task requirements such as file transfer and function calculation.

[0171] In this example, the average values ​​of response time, resource utilization, cluster load balancing, etc. obtained from ten actual runs are used as the initial service quality indicator tuple for each resource allocation. The optimization and elimination rates in the DSACO algorithm are set to 0.6 and 0.1, the number of optimizations is 60, and the optimization threshold is 0.1 to ensure the basic stability of the population.

[0172] Figure 3 The change of cluster resource utilization as the number of services increases is shown. As the number of tasks increases, the resource occupancy of the four algorithms changes accordingly. From the experimental results, it can be seen that when the number of tasks is small, the resource utilization of each algorithm is equal due to sufficient available resources in the cluster. However, as the number of tasks continues to increase, the resource utilization of the traditional polling algorithm decreases, because the polling allocation will cause a certain resource of the node to be exhausted while the utilization of other resources is very low. By comparing the MOPSO and ATNPA algorithms, it can be seen that the resource allocation strategy proposed in the present invention is similar to the heuristic algorithm in terms of resource utilization. This is mainly because the present invention takes into account task fairness, and the selected tasks have a certain impact on resource utilization; at the same time, when the expected load is too high, the elimination strategy is used to reduce tasks to ensure the reliability of the cluster.

[0173] Figure 4 The figure shows how the task completion time changes as the number of tasks increases. As can be seen from the figure, compared with other intelligent optimization algorithms, the algorithm of the present invention effectively reduces the completion time of resource allocation and balances the task execution time. As the number of tasks continues to increase, the execution time of the allocation scheme under the DSACO algorithm is less than that of other intelligent algorithms, proving that the algorithm's optimization ability in distributed cluster systems is better than other comparison algorithms.

[0174] The present invention combines fair allocation with combinatorial optimization ideas, and the proposed DSACO algorithm has a stronger ability to search for the optimal solution, which can effectively shorten the allocation time of batch tasks and save system resources. Compared with existing methods, the present invention optimizes the allocation scheme from multiple dimensions such as fairness, resource utilization, and allocation time, which can improve the rapid allocation capability of resources and improve the scalability, availability, and reliability of the system.

[0175] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A distributed cluster resource autonomous scheduling method based on DSACO, characterized in that: The method comprises the following steps: Step 1, building a system information repository RES based on node configuration information, node dynamic information and network information, and monitoring the load status information on each node in the distributed cluster and the link information between nodes in real time, and storing them in the system information repository RES; Step 2, evaluating the server nodes from the aspects of computing capacity, load condition, and resource utilization, calculating the comprehensive load and resource utilization of each node, and storing them in the system information repository RES; Step 3: Based on the heterogeneity of nodes and user tasks in the server cluster, the resource scheduling problem is modeled as a multi-objective optimization problem, and a distributed resource autonomous allocation model is constructed; Step 4: Based on the distributed resource autonomous allocation model, according to the differences in priorities and resource requirements of user tasks, a fair allocation algorithm with priority iteration is used to construct a set of allocatable tasks from all tasks within the limit of available cluster resources; Step 5: Based on the allocatable task set and cluster node status information, a cluster resource decentralized search and competition optimization algorithm DSACO is constructed, so that each task searches for a server node with high fitness according to its own resource demand characteristics, and determines a preliminary resource allocation plan; Step 6, analyzing the expected load change of nodes, executing a competitive optimization algorithm for nodes with expected load differences greater than a preset standard, and determining a resource allocation scheme; the competitive optimization algorithm includes: using a consistent hashing algorithm to first randomly assign low-priority tasks to low-load nodes, and further performing combinatorial optimization on the random assignment results based on a swarm intelligence optimization algorithm; Step 7: Use the comprehensive load and node pre-allocated task characteristics to evaluate the expected load after resource allocation. If the expected load of the node does not exceed the set threshold, resource allocation is performed; otherwise, considering the fairness and priority of resource allocation, an elimination strategy is used for further processing to reduce the number of tasks to be executed to reduce the expected load.

2. The distributed cluster resource autonomous scheduling method based on DSACO according to claim 1 is characterized in that: Step 1 builds the system information repository RES based on the node configuration information, node dynamic information and network information, specifically: Among them, RES static The static resources of the cluster include the number of CPU cores CN, the memory size of the node MS, the disk size DS, the clock frequency CF, the disk IO rate DR and the network bandwidth NB: RES dynamic It is the real-time status information of the node, which needs to be collected again each time resource allocation is performed, including CPU usage CN use , memory usage MS use , Disk Usage DS use and node bandwidth usage NB use RES network Dynamic information of the network, including average transmission delay NDE avg , the average throughput NT of the cluster where the node is located avg and the average packet loss rate NL avg .

3. The distributed cluster resource autonomous scheduling method based on DSACO according to claim 2 is characterized in that: In step 2, the comprehensive load and resource utilization of each node are calculated based on the collected node load status information and computing power; The calculation formula of the comprehensive load is: Where NWL i is the comprehensive load of the i-th node. According to the characteristics of heterogeneous cluster tasks, the load is divided into IO load NL i and calculate the load CL i Two parts; The resource utilization rate is calculated as follows: In the formula, SRU i represents the resource utilization of the i-th server node, ω j represents the weight of resource j, SRC i ={SRC i1 , SRC i2 ...SRC im } represents the resource capacity vector of the i-th server node, where m is the number of resource types, SRC ij represents the total amount of resource j in the i-th node of the server, SRL i ={SRL i1 , SRL i2 …SRL im } represents the available capacity of resources in server i, SRL ij represents the available capacity of resource j in server i.

4. The distributed cluster resource autonomous scheduling method based on DSACO according to claim 3 is characterized in that: Step 3 specifically includes: Step 3.1: Establish cluster system model MODEL system : MODEL system =(S,R,U,W) Among them, S represents the cluster node set, R corresponds to the node resource set, U represents the system user set, and W represents the user's task set; Step 3.2, establish resource scheduling optimization objectives and constraints: (1) Fairness of resource allocation: Among them, f fairness The fairness of resource allocation is evaluated by the variance of the maximum resource allocation quantity of the user, UTR u represents the maximum total amount of resource allocation for user u, represents the average of the total amount of resource allocation, n represents the number of users in the cluster, URV u ={URV u1 , URV u2 , ..., URV um } represents the resource demand vector of user u, where URV uj represents the demand of user u for resource j, UTN u represents the number of tasks of user u that has obtained resources, RL = {RL1, RL2…RL m } represents the cluster resource availability vector, RL j represents the available amount of cluster resource j, and U is the user set; Resource allocation should satisfy the requirement that the total amount of allocated resources is less than the available amount of cluster resources. The judgment method is: (2) Cluster load balancing: Among them, f loadbalance For cluster load balancing, represents the average load of the cluster, sn represents the number of server nodes, and is evaluated by the variance of the total load of each node; (3) Cluster resource utilization: Among them, f usage is the cluster resource utilization; (4) Resource scheduling time: The resource scheduling time f is determined by monitoring the time interval from the start of scheduling each task to determining the mapping relationship and submitting it to the node. time It is the maximum time interval of the currently scheduled task, and the calculation formula is: f time =max(w end -w start )w∈W In the formula, w start represents the start time of resource scheduling, i.e. the start time of task fair selection, w end Indicates the time when the mapping relationship was finally determined; Step 3.3, use the linear regression model to solve the multi-objective optimization problem of the resource scheduling process and establish a distributed resource autonomous allocation model: In the formula, ξ f ,ξ t ,ξ u ,ξ l Corresponding weights for different optimization objectives; The model optimization goal is to minimize f.

5. The distributed cluster resource autonomous scheduling method based on DSACO according to claim 4 is characterized in that: The fair allocation algorithm of priority iteration in step 4 is specifically as follows: when the total amount of task resource demand is greater than the available amount of cluster resources and all tasks cannot be allocated, a set of tasks that can currently be allocated is selected from all tasks.

6. The distributed cluster resource autonomous scheduling method based on DSACO according to claim 5 is characterized in that: Step 4 specifically includes: Step 4.1, comprehensively consider the user's task priority and the number of resources allocated to the user, and calculate the minimum resource allocator through the following formula; Among them, priority u Indicates the task priority; represents the dominant resource of user u, Indicates that user u has Quantity of demand; The calculation formula is: Step 4.2, allocating resources for the tasks of the minimum resource allocator; If there are multiple minimum resource allocators at the same time, the priority is determined and the minimum resource allocator with the highest priority is selected for allocation; Step 4.3, looping through steps 4.1 to 4.2 until the total resource requirements of the selected tasks reach the current available cluster resources, thereby determining the allocatable task set WSL; Step 4.4, increase the priority of currently unselected tasks outside the assignable task set WSL.

7. The distributed cluster resource autonomous scheduling method based on DSACO according to claim 6 is characterized in that: Step 5 specifically includes: Phase 1: Determine decision variables and construct search space: Step 5.1: construct an algorithm search space based on the characteristics of server resources and tasks; specifically, it includes: Step 5.1.1, analyze the allocatable task set WSL and determine the number of dimensions of the search space to be the number of types of resource requirements in the task requirement set; Step 5.1.2, analyze the available resources of the cluster nodes and determine the boundary value of each dimension as the maximum available amount of cluster node resources; Step 5.1.3, according to the server node resource availability vector, locate the node to the corresponding position in the search space; Step 5.2, determine the initial position of the task according to the task resource demand vector and the available resources of the cluster nodes; Step 5.3, generate the initial direction and speed of each task, the speed of task w is v, the direction is o, and the total number of iterations of the search process is δ; Phase 2: The allocation results are obtained through decentralized search, including: Step 5.4, start iterative search, calculate the fitness value of the task at the current node according to the following formula, and update the historical optimal fitness value of the task; FIT i =a T RUT i +a D RUD i +a L NWL i Where FIT i Represents the fitness value of the current task and the i-th node, RUT i Represents the resource utilization change after the i-th node allocates resources to the task w of the current user u; α T , α D and α L Respectively represent the weight parameters of the three indicators; RUT i Calculation formula: Where URV wj represents the demand of task w for resource j; RUD i Represents the deviation of the current server node resource utilization from the global average resource utilization, RUD i The calculation formula is: In the formula, SRU i’ represents the resource utilization of the i'th node of the server; Step 5.5, determine the moving direction of the next task in the search space; The optimal search direction is determined according to the task resource demand vector, and a random interference vector is introduced for interference. The search direction and speed of task w of user u at a certain moment are shown as follows: in, is the weight of the random vector, which is used to adjust the optimization ability and convergence speed. Random is a random interference vector, which consists of m random numbers κ j Composition, O is the vector of speed and direction; Step 5.6, loop the above iterative process until the final number of iterations is reached, and compare the fitness value of the current position with the historical optimal position, move the task to the node with high fitness, confirm the allocation result, and complete the preliminary resource allocation.

8. The distributed cluster resource autonomous scheduling method based on DSACO according to claim 7 is characterized in that: In step 6, the expected load change of the nodes is analyzed, and the competitive optimization algorithm is executed for the nodes with large expected load differences, specifically including: When multiple tasks eventually find the same node, if the node has sufficient resources, resources are directly allocated; otherwise, the competitive optimization algorithm is executed.

9. The distributed cluster resource autonomous scheduling method based on DSACO according to claim 8, characterized in that: Step 6 specifically includes: Step 6.1, for each server node with task conflicts, select high-priority tasks for allocation based on task priority, and then integrate unselected tasks into a task set to be optimized WCL; the task conflict refers to the task resource demand being greater than the node available resource; Step 6.2: Use the consistent hashing algorithm for random allocation, connect all nodes into a hash ring, and randomly allocate tasks in the WCL to nodes. If the node resources are insufficient, move to the next node in the specified direction. After a certain number of moves, if there are still tasks that cannot find the corresponding nodes, remove the tasks and perform priority iteration to participate in the next round of resource allocation. Step 6.3, optimize the allocation result by combining the Black Widow Optimization Algorithm to obtain an optimized resource allocation plan; specifically including: Step 6.3.1, abstract each server as an individual in the black widow population, define the optimization rate pr, elimination rate cr, optimization times on, optimization threshold ot, and the number of servers participating in the optimization N in the algorithm; The cluster nodes are sorted according to the resource utilization rate, and N×pr individuals with low resource utilization rate in the cluster are selected to participate in the optimization based on the optimization rate pr; the individuals participating in the optimization are randomly grouped for reproduction, and the reproduction process is shown in the following formula: Among them, S1 and S2 are the selected node pairs to be optimized, S ` 1 and S ` 2 is the optimized node pair, α is a randomly generated probability array of auxiliary reproduction; Step 6.3.2, calculate the fitness value of the new individuals bred, re-rank the N individuals, select N×cr individuals with high fitness in the population based on the elimination rate cr, and optimize the remaining individuals; Step 6.3.3: If the upper limit of the number of combined optimizations is reached or the optimization effect cannot reach the optimization threshold, the algorithm terminates and the optimized resource allocation plan is obtained.

10. The distributed cluster resource autonomous scheduling method based on DSACO according to claim 9, characterized in that: Step 7 specifically includes: For the resource allocation scheme obtained in step 6, calculate the expected comprehensive load and resource utilization of each node, and compare whether the expected result exceeds the node load threshold. If NWL i If the load threshold is exceeded, some tasks will be eliminated to ensure the normal operation of the cluster; Among them, the elimination strategy is as follows: That is, the priority of the task set on the slave node w , resource allocation fairness f fairness The fitness value FIT of the task at the node is comprehensively evaluated, and the task with the smallest evaluation result is eliminated.

Images