Efficient task scheduling method based on distributed collaboration

Abstract

The invention discloses an efficient task scheduling method based on distributed collaboration, and belongs to the technical field of distributed computing. Aiming at the problems of non-uniform resource allocation, insufficient task type difference adaptation, lack of multi-task dependence global optimization and the like existing in the existing scheduling strategy, the invention provides the following technical scheme: firstly, classifying tasks based on calculation characteristics and establishing a multi-dimensional resource index; secondly, performing dynamic weight evaluation on the heterogeneous resources by adopting an optimal worst weighting method (BWM), and constructing a task-resource matching matrix; a task dependency relationship is modeled through a directed acyclic graph (DAG), and a global priority sequence is generated in combination with a list scheduling algorithm; and finally, single-task optimal node matching is realized by adopting an elimination selection method, and cooperative scheduling is performed on multiple tasks by applying an arithmetic optimization algorithm (AOA). According to the method, accurate resource matching of a calculation-intensive task and a data-intensive task is realized through three technical dimensions of task feature perception, resource dynamic adaptation and dependency relationship collaborative optimization.

Description

Technical Field

[0001] The present invention relates to the field of task scheduling, and particularly to an efficient task scheduling method based on distributed collaboration, which is applicable to task scheduling optimization in a distributed system to improve the scheduling efficiency of tasks. Background Art

[0002] In a distributed architecture, a task is usually composed of multiple subtasks and executed on multiple computing nodes. However, due to the complex dependencies between subtasks and the distribution of resources on different nodes, how to efficiently schedule tasks to start execution in the shortest time, improve resource utilization, and achieve load balancing has become a core challenge.

[0003] Currently, Kubernetes (K8s), as a mainstream container orchestration tool, has been widely used in distributed computing environments. However, its existing scheduling strategies have the following limitations:

[0004] 1. The existing resource scheduling strategies are insufficient in resource matching. Usually, a unified resource allocation method is adopted, and the optimal configuration of different types of resources is not carried out. This method may lead to the underutilization of dedicated hardware resources on some nodes, while other computing nodes are difficult to undertake tasks due to insufficient resources.

[0005] 2. The differences in task types are not fully considered. In particular, the different resource requirements of compute-intensive tasks and data-intensive tasks are not effectively matched. Compute-intensive tasks usually require strong CPU computing power support, while data-intensive tasks are more dependent on high-bandwidth networks and large-capacity storage resources. Under the existing scheduling strategies, the allocation of task resources lacks pertinence, which may lead to low utilization of computing resources or storage resources.

[0006] 3. A task is usually composed of multiple interdependent subtasks, which need to be executed collaboratively on different computing nodes. However, in multi-task processing, traditional scheduling algorithms usually rely on a priority queue for sequential scheduling. The scheduling order is only based on the resource requirements of the current task and the available resources of the system, and the global requirements of the task are not fully considered. This scheduling method is less efficient in multi-task deployment, difficult to achieve global optimality, and fails to utilize the dependencies between tasks for optimization, resulting in a delay in the startup time of tasks and affecting the overall performance of the system.

[0007] The above disadvantages of the existing technologies will affect the efficiency of task scheduling, resulting in too low resource utilization and unbalanced system load. Therefore, the present invention proposes an efficient task scheduling method based on distributed collaboration. Summary of the Invention

[0008] The present invention aims to solve the technical problems of uneven resource allocation (such as underutilization of dedicated hardware), insufficient handling of task type differences (such as mismatch of resources for computing and data-intensive tasks), and lack of global optimization of multi-task dependencies (such as the inability of traditional priority queues to achieve optimal scheduling order) in task scheduling in distributed systems, thereby improving resource utilization, shortening task startup time, and achieving load balancing.

[0009] In view of the above problems, the present invention proposes an efficient task scheduling method based on distributed collaboration, which includes the following steps:

[0010] S1: Classify tasks according to computing characteristics and define different types of tasks and cluster resource indicators;

[0011] S2: Evaluate the resource usage in the cluster and use the best-worst weighting method to weight the resources required by different types of tasks;

[0012] S3: Use a directed acyclic graph to represent the dependencies between multiple tasks, and combine it with a list scheduling algorithm to prioritize the scheduling order of tasks;

[0013] S4: Based on the resource weights of different types of tasks, a single task is scheduled using the elimination and selection method. Based on the scheduling priorities of multiple tasks, an arithmetic optimization algorithm is used to schedule multiple tasks.

[0014] In the above technical solution, the specific description of S1 is:

[0015] S11: Tasks are divided into compute-intensive and data-intensive tasks based on their computing characteristics. Compute-intensive tasks require high-frequency, high-cache CPUs, such as matrix operations and image processing. Data-intensive tasks rely on memory and disk read / write capabilities as well as network performance, such as data storage and data communication.

[0016] S12: Based on the resource requirements of different types of tasks, tasks are further divided into CPU-intensive, memory-intensive, I / O-intensive, and network-intensive tasks. Resource indicators in the cluster are defined, including CPU occupancy, memory occupancy, disk occupancy, and network occupancy.

[0017] S13: Add the task type to the description file when uploading the task, and add the corresponding strategy to the task scheduling algorithm S4 to make a decision.

[0018] S131: Add a task resource-intensive type to the Kubernetes Pod description file, and perform different weight calculations on the resources of the nodes in the cluster based on the task type.

[0019] S132: Scoring the nodes in the cluster through the scheduler.

[0020] In the above technical solution, the specific description of S2 is as follows:

[0021] S21: Collect the resource information of each node in the cluster, including CPU, memory, disk, and network occupancy;

[0022] S22: Determine the optimal index c best , that is, the most important index among all indexes, and the least important index among all indexes, namely the worst index c worst ;

[0023] After obtaining the optimal and worst indexes of different types of tasks, according to the degree of dependence of different tasks on resources, score the relative importance of the optimal index and each index, and the relative importance of each index and the worst index, to obtain the BO and OW data of different types of tasks;

[0024] S231: Compare the optimal index c best with all other indexes to form a comparison vector A best = (a best1 , a best2 , …, a bestn ), T , representing the relative importance of the best index and each index, where a bestj represents the relative importance degree of index c B to index c j ;

[0025] S232: Compare all indexes with the worst index c worst to represent the relative importance of other indexes and the worst index, and form a comparison vector A worst = (a 1worst , a 2worst , …, a n2orst ), T , where a iworst represents the relative importance degree of index c i to index c worst ;

[0026] S24: Solve the optimal weight values of each index, and define (w1, w2, …, w n ) as the weights of each index. The weight allocation should theoretically satisfy formulas (1) and (2):

[0027]

[0028] Among them, w best represents the weight of the optimal index, w jDenotes the weights of all indicators except the optimal and worst indicators, a jworst Denotes the relative importance of indicator j with respect to indicator c worst ;

[0029] S241: Due to the inconsistency problem in pairwise comparison, there will be a deviation between the two ends of the equation. Therefore, the problem of finding the optimal weights for each indicator can be transformed into a mathematical optimization problem. The transformed mathematical programming problem is shown in (3):

[0030]

[0031] Where ξ represents the deviation value that appears between the two ends of the equation.

[0032] S242: Solve the above mathematical programming to obtain the weight values for different types of tasks.

[0033] S25: After solving the optimal weights (w1, w2,..., w n ) of each indicator, use the maximum absolute deviation ξ* to calculate the consistency ratio CR of the obtained weights. The formula is shown in (4):

[0034]

[0035] Where the consistency index CI is a given value, and its order is determined according to the number of indicators. The smaller the consistency ratio CR, the higher the consistency. Generally, when CR is greater than 0.5, it may be necessary to re-evaluate and judge. When it is 0, it represents complete consistency.

[0036] In the above technical solution, the specific description of S3 is as follows:

[0037] S31: Assume that G(V, E, C, R) is the dependency graph of the task. Among them, V represents the set of tasks, and each node v i corresponds to an independent task; E represents the set of dependency relationships between tasks. If there is an edge (v i , v j ) ∈ E, it means that there is data transmission between task v i and task v j , and task v j needs to rely on the data of task v i for calculation. C represents the set of data transmission amounts between tasks, where c ij represents the data transmission requirement between task v i and task v j . R records the resource consumption of the task, where R i reflects the computing resources that task v i needs to consume, including CPU requirements, memory requirements, network requirements, and disk requirements;

[0038] S32: Sort multiple tasks in combination with the Heterogeneous Critical Node First (HCNF) list scheduling algorithm;

[0039] S321: Traverse the DAG graph of task dependencies to identify the critical path CP in the DAG (the critical path is the longest path from the entry node to the exit node of the DAG, obtained through depth-first traversal). If there are multiple critical paths, randomly select one to find the critical path. The critical path is the longest path from the entry node to the exit node of the DAG, and all tasks on the critical path are regarded as critical tasks.

[0040] S322: After finding the critical path, perform hierarchical sorting according to the path. Use breadth-first traversal to calculate the level of tasks, and its formula is as shown in (5):

[0041]

[0042] Where Level(i) represents the hierarchical sorting value, pred(i) represents that task k is a predecessor task of task i, and the Level value of task i is equal to the maximum Level value of all its predecessor nodes plus 1. Tasks without predecessors and successors will be added by 1 on the maximum level of the remaining tasks.

[0043] S323: Then calculate the priority of critical tasks according to the level. Among them, tasks without predecessor nodes are called entry tasks, and their level and priority are 0. The priority calculation formula of critical tasks is as shown in (6):

[0044] Prior(i) = Level(i) (6)

[0045] Where the Prior(i) of critical tasks is equal to the level value.

[0046] S324: Nodes not on the critical path are non-critical nodes, and their scheduling order is behind, placed at the end of the entire scheduling list. The priority calculation formula is as shown in (7):

[0047]

[0048] Where key(i) represents the task number in the critical path, and the priority of non-critical tasks is obtained by adding the maximum priority of all tasks on the critical path to its own level.

[0049] S33: Sort all tasks according to the priority of the tasks to obtain the corresponding task list for determining the scheduling order. The list scheduling algorithm is used to complete the scheduling of tasks in the shortest time. The critical tasks on the critical path will be scheduled first, ensuring the start of the overall tasks. At the same time, the nodes in the dependent predecessors are also relatively ahead in scheduling, reducing the waiting time of subsequent tasks.

[0050] In the above technical solution, the specific description of S4 is as follows:

[0051] S41: When the user selects single-task scheduling, sort the nodes by the Elimination and Choice Translating Reality (ELECTRE) method and select the optimal nodes for scheduling;

[0052] S411: According to the weights obtained in step S2 and the index information of the nodes, sort the nodes according to the following steps.

[0053] S412: Standardize the decision-making indicators at different scales, use vector normalization, and use 1 - r for negative indicators ij , and its calculation formula is as shown in (8)

[0054]

[0055] where x ij represents the value of index j on node i and node respectively;

[0056] S413: Perform weighted calculation on the normalized matrix, and its calculation formula is as shown in (9):

[0057] v ij = w j ×r ij (9)

[0058] w j represents the weight value of index j;

[0059] S414: For each pair of nodes k and node 1, divide the index set J into two non-overlapping subsets. Among them, the consistent set C kl of node k with respect to node 1 is composed of the indicators in node k whose weighted scores are not lower than those of node 1, and the other is composed of the indicators in node k whose weighted scores are lower than those of node 1, which is called the inconsistent set D k1 of node k with respect to node 1.

[0060] S415: After comparing the consistent and inconsistent sets, the sum of the weights related to the consistent set is regarded as the consistency index, and its calculation formula is as shown in (10):

[0061]

[0062] S416: Calculate the inconsistency index, that is, the degree to which a node is worse than another node. Its calculation formula is shown in (11):

[0063]

[0064] where v kj and v lj represent the results of weighted calculation of index j of node k and the results of weighted calculation of index j of node l respectively; S417: Determine the consistency threshold, construct the dominance relationship matrix. When the element ckl in C kl exceeds the consistency threshold , node A k is superior to node A l in most indicators. The calculation formula of the consistency threshold is shown in (12):

[0065]

[0066] S418: Construct a Boolean-type consistency dominance matrix F, which is 1 when , otherwise 0. Each value in the matrix represents the dominance relationship of a node relative to another node. Similarly, an inconsistency dominance matrix G can be constructed. The threshold is usually set to 0.3, which is 1 when , otherwise 0.

[0067] S419: Construct the final dominance relationship matrix. This step aims to combine matrices F and G into an aggregated dominance matrix E. The element e kl in E is obtained by multiplying f kl and g kl . Its calculation formula is shown in (13):

[0068] e kl = f kl ×g kl (13)

[0069] where f kl and g kl are the elements in the consistency dominance matrix F and the inconsistency dominance matrix G respectively. f kl indicates that node k is superior to 1 in the consistency set, and g kl indicates that node k is superior to 1 in the inconsistency set.

[0070] S420: Eliminate inappropriate node solutions. In the previous definition, when , node Ak is superior to A l in most indicators. When At this time, node A k is superior to A in the inconsistent set l , that is, A k The disadvantage will not be too serious if both and are satisfied, then it is considered that solution A k is superior to A l , that is, when the element in the aggregation transcendence matrix E is 1.

[0071] S421: Sort the remaining nodes according to the net transcendence index. For the net transcendence index Q of each node k , the calculation formula is as shown in (14):

[0072]

[0073] where e ki , e ik respectively represent whether node k is superior to node i and whether node i is superior to node k;

[0074] S422: Schedule the task to the optimal node for execution through the Kubernetes scheduler.

[0075] S42: When the user selects multi-task scheduling, determine the scheduling order of multiple tasks with dependencies according to S3, combine the arithmetic optimization algorithm to give the scheduling nodes of multiple tasks, and schedule multiple tasks according to the scheduling order and scheduling nodes.

[0076] S421: Define the waiting time W of the task, and its calculation formula is as shown in (15):

[0077]

[0078] where W(i, m) is the waiting time of task i when executed on node m, W(pred, n) is the waiting time of the predecessor task pred of task i on node n, D(pred, i) is the data transfer volume from task pred to task i, which can usually be estimated by the task, and B(m, n) is the network bandwidth between node m and node n, obtained through measurement;

[0079] S422: Determine the objective function f of the arithmetic optimization according to the critical tasks obtained from S3, and its calculation formula is as shown in (16):

[0080]

[0081] S423: Randomly generate a group of candidate solutions to initialize the population of the arithmetic optimization. The dimension of each solution corresponds to the decision variable of the optimization problem. Assume that there are n computing nodes N in the cluster i ={N1, N2,..., Nn}, there are m tasks T to be deployed i ={T1, T2, …, T m}, the initialization matrix D is represented as shown in (17):

[0082]

[0083] S424: According to the constraint conditions, if the matrix meets the requirements, adjust the assignment of the matrix. The solutions in the matrix should satisfy the constraint conditions of the scheduling problem, that is, tasks that require specific resources can only be deployed on the corresponding working nodes, the same task can only appear on one node, multiple tasks can be deployed on one node, and the total demand of tasks cannot exceed the maximum available total of nodes in the cluster;

[0084] S425: Calculate the mathematical acceleration function MOA. In each iteration, AOA will determine whether the search strategy is exploration or exploitation through the MOA function, and design an inertia mechanism to dynamically adjust the choice between exploration and exploitation. In the initial stage of the algorithm, more exploration is expected, making the growth of MOA slow to reduce the probability of falling into local optimal solutions. In the later stage, more exploitation is expected, making it grow faster to accelerate the convergence speed of the algorithm and find the global optimal solution more efficiently. Its calculation formula is as shown in (18):

[0085]

[0086] S426: Iteratively select solutions by comparing the objective fitness values f of different solutions.

[0087] S427: Stop iterating when the maximum number of iterations is reached, and give the final node solution set for multiple task scheduling;

[0088] S428: According to the scheduling order and scheduling nodes, the Kubernetes scheduler schedules multiple tasks.

[0089] In summary, due to the adoption of the above technical solutions, the beneficial effects of the present invention are as follows:

[0090] 1. An efficient scheduling method based on distributed collaboration of the present invention classifies the computational characteristics of tasks and combines resource weight calculation to achieve precise matching of different types of tasks and computational resources, avoiding waste of computational resources and improving the overall efficiency of task execution.

[0091] 2. An efficient scheduling method based on distributed collaboration of the present invention innovatively uses the optimal and worst weight assignment method to perform weighted analysis on task requirements, enabling the scheduling scheme to fully consider the utilization of different computational resources, ensuring that tasks run on appropriate computational nodes, and improving the load balancing of the system.

[0092] 3. An efficient scheduling method based on distributed collaboration in the present invention innovatively proposes a collaborative scheduling algorithm based on task dependency relationships, uses a directed acyclic graph to represent task dependency relationships, and combines a list scheduling algorithm to optimize the task priority sorting, making the task scheduling more reasonable, reducing the waiting time of tasks, and improving the overall execution efficiency of tasks.

[0093] 4. An efficient scheduling method based on distributed collaboration in the present invention, for single-task scheduling, adopts the elimination and selection method to match the optimal computing nodes, and for multi-task scheduling, combines the arithmetic optimization algorithm to optimize the global scheduling scheme to ensure that tasks can be efficiently allocated to the optimal computing nodes and improve the overall computing power of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] Figure 1 is a simple flowchart;

[0095] Figure 2 The dependency graph of tasks. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0096] The following will give a detailed description of the embodiments of the present invention. Although the present invention will be described and illustrated in conjunction with some specific embodiments, it should be noted that the present invention is not limited to these embodiments only. On the contrary, any modifications or equivalent replacements made to the present invention should be covered within the scope of the claims of the present invention.

[0097] In addition, in order to better illustrate the present invention, numerous specific details are given in the following detailed description of the embodiments. Those skilled in the art will understand that the present invention can be implemented without these specific details.

[0098] Based on the above problems, the present invention proposes an efficient task scheduling method based on distributed collaboration, including the following steps:

[0099] S1: Classify tasks according to computing characteristics, and define different types of tasks and cluster resource metrics;

[0100] S2: Evaluate the resource usage in the cluster, and use the optimal and worst-weighting method to score the weights of the resources required for different types of tasks;

[0101] S3: Use a directed acyclic graph to represent the dependency relationships of multiple tasks, and combine a list scheduling algorithm to sort the scheduling order of tasks by priority;

[0102] S4: Schedule a single task using the elimination and selection method according to the resource weights of different types of tasks, and schedule multiple tasks using the arithmetic optimization algorithm according to the scheduling priorities of multiple tasks.

[0103] In the above technical solution, the specific description of S1 is as follows:

[0104] S11: Tasks are divided into compute-intensive and data-intensive tasks according to their computing characteristics. Compute-intensive tasks require high-frequency and high-cache CPUs, such as matrix operations and image processing tasks. Data-intensive tasks rely on the read / write of memory and disks as well as network performance, such as data storage and data communication tasks;

[0105] S12: According to the resource requirement characteristics of different types of tasks, tasks are further divided into CPU-intensive, memory-intensive, IO-intensive, and network-intensive tasks, and the resource metrics in the cluster are defined to include: CPU occupancy, memory occupancy, disk occupancy, and network occupancy;

[0106] S13: Add the task type to the description file when the task is uploaded, and add a corresponding policy to the task scheduling algorithm to make a decision on it.

[0107] S131: Add the task resource-intensive type to the description file of the Kubernetes Pod, and perform different weight calculations on the resources of the nodes in the cluster according to the task type;

[0108] S132: Score the nodes in the cluster through the scheduler.

[0109] In the above technical solution, the specific description of S2 is as follows:

[0110] S21: Collect the resource information of each node in the cluster, including CPU, memory, disk, and network occupancy;

[0111] S22: Determine the optimal metric c B , that is, the most important metric among all metrics, and the worst metric c worst ;

[0112] For CPU-intensive tasks, the optimal metric is CPU occupancy, as it determines the execution efficiency of CPU computing tasks. The worst metric is disk occupancy, as CPU computing tasks hardly rely on disk I / O. Memory-intensive tasks usually access and operate on data in memory frequently without involving a large number of computing or I / O operations. Therefore, the requirement for disk occupancy is not high. For I / O-intensive tasks, such as LAN FTP transfers and file read / write tasks, the optimal metric is disk occupancy, which determines the speed of I / O operations. The worst metric is CPU occupancy, as in most cases of I / O-intensive tasks, the CPU is waiting for the read / write operations of the hard disk and memory, and the CPU load is not high. For network-intensive tasks, such as online video services and Socket network transfers, in addition to network communication, disk reads and writes involved in file transfers are also relatively large, while the CPU is a resource with a low load.

[0113] S23: After obtaining the optimal and worst metrics for different types of tasks, based on the resource dependence of different tasks, score the relative importance of the optimal metric to each metric (Best-to-other, BO) and the relative importance of each metric to the worst metric (Others-to-worst, OW) to obtain the BO and OW data for different types of tasks;

[0114] S231: Compare the optimal metric c B with all other metrics to form a comparison vector A B =(a B1 , a B2 , …, a Bn ) T , representing the relative importance of the best metric to each metric, where a Bj represents the relative importance of metric c B to metric c j .

[0115] S232: Compare all metrics with the worst metric c worst to represent the relative importance of other metrics to the worst metric, and form a comparison vector A W =(a 1W , a 2W , …, a nW ) T , where a iw represents the relative importance of metric c i to metric c W .

[0116] S24: Solve the optimal weight values of each metric, and define (w1, w2, …, w n) is the weight of each indicator. Theoretically, the weight distribution should satisfy formulas (1) and (2):

[0117]

[0118] where w best Represents the weight of the optimal indicator, w j Indicates the weights of the indicators except the best and worst indicators, a jworst Indicates the relationship between index j and index c worst the relative importance of

[0119] S241: Due to the inconsistency problem in the pairwise comparison, there will be deviations on both sides of the equation. Therefore, the problem of finding the optimal weight for each indicator can be transformed into a mathematical optimization problem. The converted mathematical programming problem is shown in (3):

[0120]

[0121] Where ξ represents the deviation value appearing on both sides of the equation.

[0122] S242: Solve the above mathematical programming to obtain weight values for different types of tasks.

[0123] S25: In solving the optimal weights of each indicator (w1, w2, ..., w n ) After that, the consistency ratio CR of the weights is calculated using the maximum absolute deviation ξ*, and its formula is shown in (4):

[0124]

[0125] The consistency index (CI) is a given value, and its order is determined by the number of indicators. The smaller the consistency ratio (CR), the higher the consistency. Generally, a CR greater than 0.5 may require reassessment, while a CR of 0 indicates perfect agreement.

[0126] In the above technical solution, the specific description of S3 is:

[0127] S31: Assume G(V, E, C, R) as the task dependency graph, where V represents the task set and each node v i Corresponds to an independent task; E represents the set of dependencies between tasks. If there is an edge (v i , v j )∈E, then it means task v i and task v j There is data transmission between them, and task v j Requires dependent task v i The data is calculated, C represents the set of data transmission between tasks, where c ijRepresents task v i With task v j The data transfer requirements between them. R records the resource consumption of the task, where R i Reflects task v i The computing resources that need to be consumed, including CPU requirements, memory requirements, network requirements, and disk requirements;

[0128] S32: Sort multiple tasks by combining the Heterogeneous Critical Node First (HCNF) list scheduling algorithm;

[0129] In multi-task scheduling, Kubernetes uses the First Come First Served (FCFS) scheduling policy. The basic principle of FCFS is to allocate resources according to the arrival order of task requests. The task that arrives at the scheduler first is given priority for scheduling. Although this method is simple to implement and easy to understand, its disadvantages are also obvious. If the cluster resources are tight, the early-arriving tasks may wait for a long time due to insufficient resources of subsequent tasks, thus affecting the overall task efficiency and response time. And the execution of target tracking tasks on heterogeneous unmanned devices usually consists of multiple interdependent subtasks, and the cooperative scheduling problem of multi-task modules needs to be considered. For example, target recognition depends on the video stream data transmitted by video acquisition, target re-identification depends on the target picture data recognized by each unmanned device, and a message queue for transmission needs to be built before data transmission and other dependencies. These tasks need to be executed cooperatively on different computing nodes, but the existing scheduling policies often consider the resource requirements of each task independently and fail to fully utilize the dependencies between tasks for global optimization. Their computing and communication capabilities vary greatly, resulting in differences in the execution and communication times of tasks on different nodes.

[0130] S321: Traverse the DAG graph of task dependencies, identify the critical path CP in the DAG (the critical path is the longest path from the entry node to the exit node of the DAG, obtained through depth-first traversal). If there are multiple critical paths, randomly select one to find the critical path. The critical path is the longest path from the entry node to the exit node of the DAG, and all tasks located on the critical path are regarded as critical tasks.

[0131] S322: After finding the critical path, perform hierarchical sorting according to the path, and use breadth-first traversal to calculate the levels of tasks. The formula is as shown in (5):

[0132]

[0133] Where Level(i) represents the hierarchical sorting value, pred(i) indicates that task k is the predecessor task of task i, and the Level value of task i is equal to the maximum Level value of all its predecessor nodes plus 1. Tasks without predecessors and successor nodes will have the maximum level of the remaining tasks increased by 1.

[0134] S323: Calculate the priority of the key task based on the hierarchy. The task without a predecessor node is called the entry task, and its hierarchy and priority are 0. The priority calculation formula of the key task is shown in (6):

[0135] Prior(i)=Level(i) (6)

[0136] The Prior(i) of the key task is equal to the level value.

[0137] S324: Nodes that are not on the critical path are non-critical nodes. Their scheduling order is later and they are placed at the end of the entire scheduling list. The calculation formula for their priority is shown in (7):

[0138]

[0139] Where key(i) represents the task number in the critical path, and the priority of a non-critical task is obtained by adding the maximum priority of all tasks on the critical path to its own level.

[0140] S33: Sort all tasks according to their priorities to obtain a corresponding task list, which is used to determine the scheduling order. The list scheduling algorithm is used to complete the scheduling of tasks in the shortest time. Critical tasks on the critical path will be scheduled first, ensuring the startup of the overall task. At the same time, nodes that depend on the predecessor are also relatively advanced during scheduling, reducing the waiting time for subsequent tasks.

[0141] In the above technical solution, the specific description of S4 is:

[0142] S41: When the user selects single-task scheduling, the nodes are sorted using the Elimination and Choice Translating Reality (ELECTRE) method, and the optimal node is selected for scheduling.

[0143] S411: Sort the nodes according to the subsequent steps based on the weights obtained in step S2 and the node index information

[0144] S412: Standardize the decision indicators at different scales using vector normalization and 1-r for negative indicators. ij , and its calculation formula is shown in (8)

[0145]

[0146] where x ij represents the value of index j on node i and node respectively;

[0147] S413: Perform weighted calculation on the normalized matrix, and its calculation formula is as shown in (9):

[0148] v ij = w j × r ij (9)

[0149] w j represents the weight value of index j;

[0150] S414: For each pair of nodes k and node 1, divide the index set J into two non - overlapping subsets. Among them, the consistent set C kl of node k with respect to node 1 is composed of the indices in node k whose weighted scores are not lower than those of node 1, and the other is composed of the indices in node k whose weighted scores are lower than those of node 1, which is called the inconsistent set D kl of node k with respect to node 1.

[0151] S415: After comparing the consistent and inconsistent sets, the sum of the weights related to the consistent set is regarded as the consistency index, and its calculation formula is as shown in (10):

[0152]

[0153] S416: Calculate the inconsistency index, that is, the degree to which one node is worse than another node, and its calculation formula is as shown in (11):

[0154]

[0155] where v kj and v lj represent the results of weighted calculation of index j of node k and the results of weighted calculation of index j of node 1 respectively; S417: Determine the consistency threshold, construct a dominance relation matrix. When the element c kl in C kl exceeds the consistency threshold node A k is superior to node A l in most indices, and the calculation formula of the consistency threshold is as shown in (12):

[0156]

[0157] S418: Construct a Boolean - type consistency dominance matrix F. When 1 when , otherwise 0, each value in the matrix represents the transcendence relationship of one node relative to another node. Similarly, the inconsistency transcendence matrix G can be constructed, and the threshold Usually set to 0.3, 1 if yes, 0 otherwise.

[0158] S419: Construct the final transcendental relation matrix. This step aims to combine the F and G matrices into an aggregate transcendental matrix E. The element e in E is kl By f kl and g kl Multiply them together and the calculation formula is shown in (13):

[0159] e kl =f kl ×g kl (13)

[0160] where f kl and g kl are the elements in the consistency transcendental matrix F and the inconsistency transcendental matrix G, respectively, kl Indicates that node k is better than l, g in the consistency set kl Indicates that node k is better than 1 on the inconsistent set.

[0161] S420: Eliminate unsuitable node solutions. In the previous definition, When node A k Better than A in most indicators l , When node A k Better than A on inconsistent sets l , that is, A k The disadvantage will not be too serious if both and It is considered that plan A k Better than A l , that is, when the element in the aggregate transcendental matrix E is 1.

[0162] S421: Sort the remaining nodes according to the net surpassing index, and for each node, the net surpassing index Q k The calculation formula is shown in (14):

[0163]

[0164] where e ki 、e ik Respectively indicate whether node k is better than node i and whether node i is better than node k;

[0165] S422: Schedule the task to the optimal node for execution through the Kubernetes scheduler.

[0166] S42: When the user selects multi-task scheduling, determine the scheduling order of multiple tasks with dependencies according to S3, combine the arithmetic optimization algorithm to give the scheduling nodes of multiple tasks, and schedule the multi-tasks according to the scheduling order and scheduling nodes.

[0167] S421: Define the waiting time W of the task, and its calculation formula is shown in (15):

[0168]

[0169] Where W(i,m) is the waiting time for task i to execute on node m, W(pred,n) is the waiting time for the predecessor task pred of task i on node n, D(pred, i) is the data transfer volume from task pred to task i, which can usually be estimated by the task, and B(m, n) is the network bandwidth between node m and node n, obtained by measurement;

[0170] S422: Determine the objective function f of arithmetic optimization according to the critical tasks obtained in S3, and its calculation formula is shown in (16):

[0171]

[0172] S423: Randomly generate a group of candidate solutions to initialize the population of arithmetic optimization. The dimension of each solution corresponds to the decision variable of the optimization problem. Assume that there are n computing nodes N i ={N1, N2,..., N n} in the cluster, and there are m tasks T i ={T1, T2,..., T m} to be deployed. Initialize the matrix D as shown in (17):

[0173]

[0174] S424: Adjust the assignment of the matrix according to the constraint conditions if the matrix meets the requirements. The solutions in the matrix should satisfy the constraint conditions of the scheduling problem, that is, tasks that require specific resources can only be deployed on the corresponding working nodes, the same task can only appear on one node, multiple tasks can be deployed on one node, and the total demand of tasks cannot exceed the maximum available total of nodes in the cluster;

[0175] S425: Calculate the mathematical acceleration function MOA. In each iteration, the AOA determines whether the search strategy is exploration or exploitation through the MOA function. Design an inertia mechanism to dynamically adjust the choice between exploration and exploitation. At the initial stage of the algorithm, more exploration is expected, so that the growth of MOA is slow, reducing the probability of falling into local optimal solutions. In the later stage, more exploitation is expected, so that it grows faster, accelerating the convergence speed of the algorithm and finding the global optimal solution more efficiently. Its calculation formula is shown in (18):

[0176]

[0177] S426: Iteratively select solutions by comparing the objective fitness values f of different solutions.

[0178] S427: Stop iterating when the maximum number of iterations is reached, and give the final set of node solutions for multiple task schedules;

[0179] S428: According to the scheduling order and scheduling nodes, the Kubernetes scheduler schedules multiple tasks.

[0180] The present invention will be further described below in conjunction with specific embodiments.

[0181] 1. The optimal and worst indexes of different types of tasks are shown in the following table:

[0182] Task type Optimal index (Best) Worst index (worst) CPU-intensive CPU occupancy rate Disk occupancy rate Memory-intensive Memory occupancy rate Disk occupancy rate IO-intensive Disk occupancy rate CPU occupancy rate Network-intensive Network occupancy rate CPU occupancy rate

[0183] Among them, the importance degree can be represented by a number between 1 and 9. Taking the comparison between the optimal index and other indexes as an example, the specific meanings are shown in the following table:

[0184] Scale value Meaning 1 <![CDATA[c B with c j Equally important<!-- 11 --> ]]> 3 <![CDATA[c B with c j Slightly more important]]> 5 <![CDATA[c B is significantly more important than c j significantly more important]]> 7 <![CDATA[c B with c j Strongly important]]> 9 <![CDATA[c B with c j Extremely important]]> 2、4、6、8 Intermediate value

[0185] Score the relative importance of the optimal index to each index (Best-to-other, BO) and the relative importance of each index to the worst index (Others-to-worst, OW) according to the scale value and the dependence degree of different tasks on resources, and obtain the BO and OW data of different types of tasks as shown in the following table:

[0186]

[0187] In the table, CPU, memory, disk, and network respectively represent their occupancy rate indicators. The BO data represents the relative importance of the optimal indicator and all indicators. For example, for CPU-intensive tasks in the table, the BO scale value of the CPU occupancy rate indicator is 1, which means the importance of the optimal indicator for CPU occupancy is 1, that is, equally important. The OW data represents the relative importance of all indicators to the worst indicator. For example, for network-intensive tasks in the table, the relative importance of the network occupancy rate indicator to the worst indicator is 5, indicating that the network occupancy rate is significantly more important than the worst indicator.

[0188] The weight values for different types of tasks obtained by solving the mathematical programming of formula (3) are shown in the following table. The weight values in the table are expressed in percentages (%).

[0189]

[0190] The consistency test is performed on the weight values of different types of tasks calculated by the BWM weight assignment method, and the consistency test results are shown in the following table:

[0191]

[0192] The above table shows the consistency ratio of the weights calculated by the BWM weight assignment method for each type of task, and the maximum absolute deviation ξ * Calculated when solving the mathematical programming by BWM, the standard for testing CR is 0.5. If it is less than 0.5, it means passing the consistency test, otherwise it means not passing the consistency test.

[0193] 2. Suppose there are tasks t1 to t6 with the following dependency relationships, and the dependency graph of the tasks is as follows Figure 2 As shown, the critical path in the figure is CP{t1, t2, t4, t6}, and the critical tasks are t1, t2, t4, t6. The task scheduling order list obtained according to the list algorithm is shown in the following table:

[0194] Task number Priority (Prior) Whether it is a critical task 1 0 Yes 2 1 Yes 3 4 No 4 2 Yes 5 5 No 6 3 Yes

[0195] In summary, the present invention has the following characteristics:

[0196] The present invention realizes the precise matching of task requirements and node resources by subdividing tasks into CPU-intensive, memory-intensive, IO-intensive, and network-intensive according to computing characteristics and combining dynamic weight calculation of resource indicators. This classification mechanism combines a resource weight optimization strategy, effectively avoiding problems such as idle dedicated hardware or resource contention caused by resource misallocation in traditional scheduling, and significantly improving the utilization rate of computing resources and task execution efficiency.

[0197] The present invention innovatively uses the Best-Worst Method (BWM) to dynamically allocate weights to heterogeneous resources. By constructing a best / worst index comparison matrix and a consistency check mechanism, it ensures that the weight calculation process conforms to the differences in task characteristics. This method overcomes the limitations of traditional unified allocation strategies, enables scheduling decisions to fully adapt to the resource sensitive points of different types of tasks, and achieves load balancing of cluster nodes while improving the utilization rate of core resources such as CPU and memory.

[0198] The present invention proposes a task dependency modeling based on a directed acyclic graph (DAG) and a heterogeneous critical node first (HCNF) list scheduling algorithm. Through critical path identification and hierarchical priority sorting mechanisms, it globally optimizes the task scheduling order. This solution solves the task blocking problem caused by the traditional FCFS strategy. By pre-positioning critical tasks and optimizing the dependency chain, it significantly shortens the overall startup waiting time of tasks, especially suitable for distributed collaborative scenarios with complex data dependencies.

[0199] The present invention designs an ELECTRE node selection algorithm and an arithmetic optimization algorithm (AOA) for single-task and multi-task scheduling respectively. Combining dynamic resource weights and task priorities, it realizes multi-level scheduling optimization. The ELECTRE method accurately selects the optimal node through a consistency / inconsistency dominance matrix, while the AOA dynamically balances global exploration and local exploitation based on a mathematical acceleration function, collaboratively reducing the overall waiting time of multi-task scheduling and improving the system throughput capacity.

[0200] The present invention extends the Kubernetes scheduler strategy, embeds task type tags into the Pod description file, and constructs an integrated framework for weight calculation, priority sorting, and optimized scheduling. While being compatible with the existing container orchestration ecosystem, this integrated solution significantly improves the deficiencies of the K8s native scheduler in heterogeneous resource adaptation, task dependency optimization, and global efficiency, and has high engineering practicability.

Claims

1. An efficient task scheduling method based on distributed collaboration, characterized in that, Including: Step S1: Classify tasks into CPU-intensive, memory-intensive, IO-intensive, and network-intensive according to computing characteristics, and define the cluster resource metric set J = {CPU occupancy rate, memory occupancy rate, disk occupancy rate, network occupancy rate}; Step S2: Based on the optimal-worst weight assignment method, calculate the weights of the resource demand indicators for different types of tasks, including: determining the optimal and worst indicators, and constructing the optimal-other comparison vector A B and the indicator-worst comparison vector A W , solve the weights of each resource indicator through mathematical programming, and verify the consistency ratio CR; Step S3: Use a directed acyclic graph to represent the multi-task dependency relationship, combine with the heterogeneous critical node priority list scheduling algorithm, calculate the task level Level(i) and priority Prior(i), and generate a task scheduling order list; Step S4: For single-task scheduling, sort and match the computing nodes based on the elimination and selection method; for multi-task scheduling, combine the arithmetic optimization algorithm and the task waiting time model to optimize the global scheduling node allocation of multi-tasks.

2. The efficient task scheduling method based on distributed collaboration according to claim 1, wherein In step S1, defining different types of tasks and cluster resource metrics includes the following steps: S11: Tasks are divided into compute-intensive and data-intensive tasks according to computing characteristics. Compute-intensive tasks require a high-frequency, high-cache CPU and include matrix operations and image processing tasks. Data-intensive tasks rely on memory and disk read / write as well as network performance and include data storage and data communication tasks; S12: According to the resource requirement characteristics of different types of tasks, tasks are further divided into CPU-intensive, memory-intensive, IO-intensive, and network-intensive tasks, and the cluster resource metric set J is defined to include: CPU occupancy rate, memory occupancy rate, disk occupancy rate, and network occupancy rate; S13: Add the task type to the description file when the task is uploaded, and add the corresponding policy in the task scheduling algorithm S4 to make a decision on it; S131: Add the task resource intensive type to the description file of the Kubernetes Pod, and calculate different weightings for the resources of the nodes in the cluster according to the task type; S132: Score the nodes in the cluster through the scheduler.

3. An efficient task scheduling method based on distributed collaboration according to claim 1, characterized in that In step S2, the steps for weighting and scoring the sources required for different types of tasks are as follows: S21: Collect the resource information of each node in the cluster, including CPU, memory, disk, and network occupancy; S22: Determine the optimal metric c according to the resource requirement characteristics of different tasks best , that is, the most important metric among all metrics, and the least important metric among all metrics, i.e., the worst metric c worst ; S23: After obtaining the optimal and worst metrics for different types of tasks, according to the degree of dependence of different tasks on resources, score the relative importance BO of the optimal metric and each metric, as well as the relative importance OW of each metric and the worst metric, to obtain the BO and OW data for different types of tasks; S231: Compare the optimal metric c best with all other metrics to form a comparison vector A of the optimal and other metrics best =(a best1 , a best2 , …, a bestn ) T , representing the relative importance of the best metric to each metric, where a bestj represents the relative importance of metric c B to metric c j ; S232: Compare all the indicators with the worst indicator c worst to represent the relative importance of other indicators and the worst indicator, and form a comparison vector A of other and the worst indicators worst =(a 1worst , a 2worst , …, a n2orst ) T , where a iworst represents the relative importance of indicator c i to indicator c worst ; S24: Solve the optimal weight values of each index, and define (w1, w2,..., w n ) as the weights of each index. The weight allocation should theoretically satisfy formulas (1) and (2): where w best represents the weight of the optimal index, w j represents the weights of the remaining indices except for the optimal and worst indices, a jworst represents the relative importance degree of index j with respect to index c worst ; S241: Due to the inconsistency problem in pairwise comparison, there will be a deviation between the two ends of the equation. Therefore, the problem of finding the optimal weight for each metric can be transformed into a mathematical optimization problem, and the transformed mathematical programming problem is as shown in (3): where ξ represents the deviation value that appears at both ends of the equation; S242: Solve the above mathematical programming to obtain the weight values for different types of tasks; S25: After solving the optimal weights (w1, w2,..., w n ), calculate the consistency ratio CR of the obtained weights using the maximum absolute deviation ξ*, and its formula is as shown in (4): where the consistency index CI is a given value, and its order is determined according to the number of metrics. The smaller the consistency ratio CR, the higher the consistency. When CR is greater than 0.5, re-evaluate and judge. When it is 0, it represents complete consistency.

4. An efficient task scheduling method based on distributed collaboration according to claim 1, characterized in that, In step S3, the detailed description of the priority sorting of the task scheduling order is as follows: S31: Use G(V, E, C, R) as the dependency graph of the task, where V represents the set of tasks, and each node v i corresponds to an independent task; E represents the set of dependency relationships between tasks. If there is an edge (v i , v j ) ∈ E, it means that there is data transmission between task v i and task v j , and task v j needs to rely on the data of task v i for calculation. C represents the set of data transmission volumes between tasks, where c ij represents the data transmission requirement between task v i and task v j . R records the resource consumption of the task, where R i reflects the computing resources that task v i needs to consume, including CPU requirements, memory requirements, network requirements, and disk requirements; S32: Combining heterogeneous key node priority list scheduling algorithm to sort multiple tasks; S321: Traverse the DAG graph of task dependencies and identify the critical path CP in the DAG. If there are multiple critical paths, randomly select one to find the critical path. The critical path is the longest path from the entry node to the exit node of the DAG graph. All tasks on the critical path are considered critical tasks. S322: After finding the critical path, perform hierarchical sorting according to the path and use breadth-first traversal to traverse the hierarchy of computing tasks. The formula is shown in (5): Where Level(i) represents the hierarchical ranking value, pred(i) indicates that task k is a predecessor task of task i. The Level value of task i is equal to the maximum Level value of all its predecessor nodes plus 1. Tasks without predecessors and successors will have the maximum level of the remaining tasks plus 1. S323: Calculate the priority of the key task based on the hierarchy. The task without a predecessor node is called the entry task, and its hierarchy and priority are 0. The priority calculation formula of the key task is shown in (6): Prior(i)=Level(i) (6) The Prior(i) of the key task is equal to the level value; S324: Nodes that are not on the critical path are non-critical nodes. Their scheduling order is later and they are placed at the end of the entire scheduling list. The calculation formula for their priority is shown in (7): Where key(i) represents the task number in the critical path, and the priority of the non-critical task is obtained by adding the maximum priority of all tasks on the critical path to its own level; S33: Sort all tasks according to their priorities to obtain a corresponding task list, which is used to determine the scheduling order. The list scheduling algorithm is used to complete the scheduling of tasks in the shortest time. Critical tasks on the critical path will be scheduled first, ensuring the startup of the overall task. At the same time, nodes that depend on the predecessor are also relatively advanced during scheduling, reducing the waiting time for subsequent tasks.

5. An efficient task scheduling method based on distributed collaboration according to claim 1, characterized in that The detailed steps of the improved task scheduling mechanism in S4 are as follows: S41: When the user selects single-task scheduling, the nodes are sorted by elimination and selection conversion method, and the optimal node is selected for scheduling; S411: The weight and node index information obtained in step S2; S412: Standardize the decision-making metrics at different scales using vector normalization. For negative metrics, use 1 - r ij , and its calculation formula is as shown in (8) where x ij represents the value of index j on node i and the node respectively; S413: Perform weighted calculation on the normalized matrix. The calculation formula is shown in (9): v ij = w j × r ij (9) w j represents the weight value of index j; S414: For each pair of nodes k and node 1, divide the index set J into two non-overlapping subsets, where the consistent set C k1 of node k with respect to node l is composed of the indices in node k whose weighted scores are not lower than those of node 1, and the other is composed of the indices in node k whose weighted scores are lower than those of node l, which is called the inconsistent set D kl ; S415: After comparing the consistent and inconsistent sets, the sum of the weights associated with the consistent sets is considered as the consistency index, which is calculated as shown in (10): S416: Calculate the inconsistency index, that is, the degree to which a node is worse than another node. The calculation formula is shown in (11): where v kj and v lj represent the results of the weighted calculation of index j of node k and the results of the weighted calculation of index j of node 1, respectively; S417: Determine the consistency threshold, construct a dominance relation matrix. When the element c kl in C kl exceeds the consistency threshold at this time, node A k is superior to node A l in most metrics. The calculation formula for the consistency threshold is shown in (12): S418: Construct a consistency transcendence matrix F of Boolean type, which is 1 when and 0 otherwise. Each value in the matrix represents the transcendence relationship of one node relative to another node. Construct an inconsistency transcendence matrix G, and the threshold is usually set to 0.3, which is 1 when and 0 otherwise; S419: Construct the final dominance relation matrix. This step aims to combine matrices F and G into an aggregated dominance matrix E, where the element e in E kl is obtained by multiplying f kl and g kl , and its calculation formula is shown in (13): e kl = f kl × g kl (13) where f kl and g kl are elements in the consistency transcendence matrix F and the inconsistency transcendence matrix G respectively, and f kl means that node k is superior to 1 in the consistency set, and g kl means that node k is superior to 1 in the inconsistency set; S420: Eliminate inappropriate node solutions. In the previous definition, when the node Ak is superior to A in most metrics l , when the node A k is superior to A in the inconsistent set l , if both and are satisfied, then the solution A k is considered superior to A l , that is, when the element in the aggregation transcendence matrix E is 1; S421: Sort the remaining nodes according to the net transcendence index. For the net transcendence index Q of each node k The calculation formula is as shown in (14): where e ki and e ik represent whether node k is superior to node i and whether node i is superior to node k, respectively S422: Schedule the task to the optimal node for execution through the Kubernetes scheduler; S42: When the user selects multi-task scheduling, the scheduling order of multiple tasks with dependencies is determined according to S3, and the scheduling nodes of the multiple tasks are given in combination with the arithmetic optimization algorithm. The multiple tasks are scheduled according to the scheduling order and the scheduling nodes; S421: Define the waiting time W of the task, and its calculation formula is shown in (15): Where W(i,m) is the waiting time for task i to execute on node m, W(pred,n) is the waiting time for the predecessor task pred of task i on node n, D(pred, i) is the data transfer volume from task pred to task i, which can usually be estimated by the task, and B(m,n) is the network bandwidth between node m and node n, obtained by measurement; S422: Determine the objective function f of arithmetic optimization according to the critical tasks obtained in S3, and its calculation formula is shown in (16): Among which W i represents the waiting time of task i; S423: Randomly generate a set of candidate solutions to initialize the population of arithmetic optimization. The dimension of each solution corresponds to the decision variable of the optimization problem. Assume that there are n computing nodes N in the cluster i = {N1, N2, …, N n}, and there are m tasks T to be deployed i = {T1, T2, …, T m}, and the initialization matrix D is represented as shown in (17): S424: According to the constraint conditions, if the matrix meets the requirements, adjust the assignment of the matrix. The solutions in the matrix should satisfy the constraint conditions of the scheduling problem, that is, tasks that require specific resources can only be deployed on the corresponding working nodes, the same task can only appear on one node, multiple tasks can be deployed on one node, and the total demand of tasks cannot exceed the maximum available total of nodes in the cluster; S425: Calculate the mathematical acceleration function MOA. In each iteration, the arithmetic optimization algorithm AOA will determine whether the search strategy is exploration or exploitation through the MOA function, and design an inertia mechanism to dynamically adjust the choice of exploration and exploitation. In the initial stage of the algorithm, more exploration is carried out to make the growth of MOA slow and reduce the probability of falling into local optimal solutions. In the later stage, more exploitation is carried out to make its growth faster, accelerate the convergence speed of the algorithm, and find the global optimal solution more efficiently. Its calculation formula is shown in (18): Where Max and Min are the maximum and minimum values of the acceleration function, which are 0.2 and 1 respectively, and t is the current iteration number, ranging from 1 to the maximum iteration number T; S426: Iteratively select solutions by comparing the objective fitness values f of different solutions; S427: Stop the iteration when the maximum iteration number is reached, and give the final node solution set for multiple task scheduling; S428: According to the scheduling order and scheduling nodes, the Kubernetes scheduler schedules multiple tasks.

Images