Big data task scheduling method and system

Abstract

The present application provides a big data task scheduling method and system, relating to the technical field of big data, the method comprising: obtaining task parameters and inter-task dependency relationship, and calculating task priority score; determining the fitness of the task and the cluster according to the data locality coefficient of the task, the cluster parameters and the cluster correlation correction coefficient; determining the target cluster of the task and the task order according to the task priority score and the fitness of the task and the cluster; triggering a fuse when the number of task failures exceeds the threshold value, and if the cross-cluster dependency fails, migrating the dependent task to the current cluster and recalculating the fitness, and restoring the task priority after consecutive success; through dynamic adjustment of task priority, optimization of cluster resource allocation, establishment of fuse and migration mechanism and other means, the core problems such as thread waste, priority failure and stability risk in traditional technology are solved, thereby improving the service stability and processing efficiency of data middle platform in complex scenarios.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, and in particular to a task scheduling method and system for big data. BACKGROUND

[0002] In the current rapid development of big data technology, multi-cluster architecture (such as Hadoop, Kubernetes cluster) has become the core deployment mode for data center to process massive data due to its strong scalability and flexibility. However, this architecture faces many challenges in task scheduling process, and the limitations of existing technologies are gradually highlighted, mainly in the following key aspects:

[0003] In a multi-cluster environment, some tasks often fail to synchronize data due to external factors such as cluster abnormalities, network fluctuations, and resource competition. Traditional scheduling mechanisms rely on fixed retry strategies, and synchronization threads are occupied for a long time during multiple retries. This not only causes inefficient consumption of thread resources, but also causes significant increase in overall synchronization delay, leading to backlog of subsequent tasks, seriously affecting the real-time performance of data processing and the throughput of the system. For example, when a cluster experiences a temporary network interruption, a large number of tasks that rely on that cluster will block threads due to frequent retries, preventing other normal tasks from obtaining resources in time.

[0004] Traditional task retry strategies are mostly based on fixed time intervals (such as exponential backoff algorithm), which can only control the execution frequency of tasks, but completely ignore the historical behavior characteristics of tasks (such as failure times, synchronization time consumption, dependency relationships, etc.). This "one-size-fits-all" approach makes it impossible for the scheduling system to dynamically adjust the priority of tasks based on their actual state. For example, for high-priority tasks that frequently retry due to cross-cluster dependency failures, traditional strategies cannot allocate more resources or adjust the scheduling order, while the inefficient retries of low-priority tasks may occupy limited system resources, resulting in low scheduling efficiency and lack of flexibility.

[0005] In high-concurrency scenarios, the disordered retries of a large number of abnormal tasks can easily cause thread pool depletion, resource deadlock, and other serious problems. When the thread pool is filled with retry requests from failed tasks, the data center will be unable to process new task requests, resulting in unavailable services and even system crashes. In addition, task failures due to cross-cluster dependencies can trigger a chain reaction, and abnormalities in one cluster can spread to multiple clusters through dependencies, further exacerbating system instability. For example, during a large promotion on an e-commerce platform, if order synchronization tasks frequently retry due to a cluster failure, the entire data center may be paralyzed, affecting real-time statistics of transaction data and business decision-making.

[0006] The existing big data task scheduling technology lacks dynamic adaptability and intelligence in a multi-cluster environment, and is difficult to meet the requirements of data middle platform for high stability and high reliability. SUMMARY

[0007] Based on the above problems, the big data task scheduling method and system of the present application solves the core problems of thread waste, priority failure and stability risk in traditional technology by combining the scheduling method of multi-dimensional information such as task parameters, dependency relationship and cluster state, dynamically adjusting task priority, optimizing cluster resource allocation, establishing a fuse and migration mechanism, etc., thereby improving the service stability and processing efficiency of data middle platform in complex scenarios.

[0008] The object of the present application is achieved by the following technical solutions:

[0009] In a first aspect, the present application provides a big data task scheduling method, which comprises:

[0010] obtaining task parameters and inter-task dependency relationships, and calculating task priority scores;

[0011] determining the fitness of the task and the cluster according to the data locality coefficient of the task, the cluster parameter and the cluster correlation correction coefficient;

[0012] determining the target cluster of the task and the task order according to the task priority score and the fitness of the task and the cluster;

[0013] triggering a fuse when the number of task failures exceeds a threshold value, and if the dependency task is migrated to the current cluster and the fitness is recalculated due to cross-cluster dependency failure, the task priority is restored after continuous success.

[0014] Preferably, the task parameters include the synchronization time difference of the task, the synchronization success flag, the number of task failures, the synchronization period, the synchronization time consumption and the current state of the task; obtaining the task parameters and the inter-task dependency relationships, and calculating the task priority scores, comprises:

[0015] mapping and normalizing each parameter of the task, and obtaining the initial priority score of the task by weighted average normalization processing of each parameter;

[0016] determining the correction coefficient according to the inter-task dependency relationship; and obtaining the first priority score of the task according to the correction coefficient;

[0017]

[0018] wherein, is the first priority score of the task; is the initial priority score of the task; is the correction coefficient, .

[0019] Preferably, the cluster parameters include cluster network latency, cluster CPU load, and memory utilization; determining the task-cluster fitness based on the task's data locality coefficient, cluster parameters, and cluster correlation correction coefficient includes:

[0020] Based on the task's locality coefficient and cluster parameters, the basic fitness of the task and the cluster is obtained.

[0021]

[0022] w1, w2, and w3 are the second weighting coefficients, w1+ + =1;

[0023] Determine the priority weight coefficient based on task priority. Combined with cluster correlation correction coefficient Determine the final fitness of the task and the cluster. :

[0024]

[0025] The cluster correlation correction coefficient is determined based on the network bandwidth between clusters, the frequency of data interaction between clusters, and the historical task scheduling success rate.

[0026] Preferably, the step of determining the target cluster and task ordering based on task priority scoring and task-cluster fitness includes:

[0027] If a task's priority score is below the score threshold, and the task's final fitness with a given cluster is greater than b1 times the task's fitness with its current cluster, then that cluster is chosen as the target cluster for the task; b1 1.2;

[0028] Calculate the final priority score (SC) of the task in the target cluster based on the task parameters and the target cluster locality gain coefficient. new ;

[0029]

[0030] The target cluster gain coefficient; ; The task is given a first priority score; among which, the target cluster gain coefficient is determined based on the localization ratio of the data required by the task, the target cluster CPU utilization, and the target cluster memory utilization.

[0031] If the final task priority score of the task in the target cluster is greater than b2 times the first task priority score of the task in the original cluster, then the task will be moved to the target cluster.

[0032] If the final task priority score of the task in multiple target clusters is greater than b² times the first priority score of the task in the original cluster, then the cluster with the highest final fitness between the task and the cluster will be selected for migration; and the task will be sorted according to its final priority score in the task list of the final target cluster; b² 1.2.

[0033] Preferably, the method further includes:

[0034] When multiple tasks are migrated to the same cluster, they are sorted according to the following rules:

[0035] All migration tasks are scored according to their final priority. Sort in descending order; if If the data locality coefficients are the same, sort them in descending order by task arrival time; if the data locality coefficients of the tasks are still the same, sort them in ascending order by task arrival time.

[0036] Before migration, check the resource capacity of the target cluster. Migration will only be performed if the migration conditions are met. The migration conditions are:

[0037]

[0038] This is the set of tasks that will be migrated to the target cluster. The resource requirement of task i in the task set; This is the safety margin factor, ranging from [0.1, 0.3].

[0039] Preferably, the method further includes:

[0040] Pre-screen the set of migration tasks and mark them as mandatory single-cluster tasks;

[0041] For tasks that are forced to be clustered, they are scored according to their final priority. Arrange in descending order to generate independent queues;

[0042] Tasks in the generated independent queue are submitted to their unique target cluster first. If the migration conditions are met, resources are immediately allocated and locked; otherwise, a cross-cluster resource coordination protocol is triggered to prioritize the resources of tasks in the degraded queue.

[0043] Preferably, the method further includes:

[0044] If a task has at least one other target cluster besides the optimal target cluster that satisfies the migration conditions, then that task is marked as Ta. multi ;

[0045] Initialize the candidate cluster list and sort it in descending order of the final fitness of each cluster and task;

[0046] Computation task Ta multi The fitness of the task with the optimal cluster and the difference between the maximum fitness of the task and that of other clusters;

[0047]

[0048] in, The fitness of this task with the optimal cluster; This is to maximize the fitness of the task compared to other clusters;

[0049] according to Sort in descending order, and prioritize processing tasks with large fitness differences within the target cluster;

[0050] Determine if the remaining resources of the optimal cluster can meet the current task; if so, assign the migration task to the optimal cluster and lock the migration eligibility of the task in other candidate clusters; if not, try the second-best cluster.

[0051] After the task is migrated to the suboptimal cluster, the resource status of the optimal cluster is continuously monitored; if the following conditions are met:

[0052] If the resource utilization rate of the optimal cluster is less than the utilization threshold for several consecutive periods, the optimal cluster migration conditions are met, and the ratio of the final fitness of the task to the optimal cluster to the final fitness of the task to the current cluster is greater than b1*b3, and the task stays in the suboptimal cluster for a duration ≥ b4 × synchronization period, then a secondary migration is triggered, where b3 ≥ 1.1 and b4 ≥ 2.

[0053] The priority weight for secondary migration is set as follows:

[0054]

[0055] For secondary migration priority weights, Assign a final priority score to the task within the current suboptimal cluster; The fitness of this task with the optimal target cluster; The fitness of this task with the current suboptimal cluster;

[0056] When the resource gap of multiple consecutive candidate clusters exceeds the preset proportion of their idle resources, a cross-cluster resource coordination request is triggered.

[0057] Preferably, the method further includes:

[0058] perform global resource rebalancing every k scheduling periods;

[0059] apply priority decay to tasks that have been stuck in suboptimal clusters for more than m periods:

[0060]

[0061] where 0<b5<1; final priority score of the task in the suboptimal cluster; final priority score of the task in the suboptimal cluster;

[0062] trigger cross-cluster dependency reorganization, re-evaluate the dependency relationship and scheduling strategy of the task and the cluster when the decayed priority is lower than the migration threshold.

[0063] Preferably, the method further comprises:

[0064] linearly scale up the local resource quota of the demoted queue tasks when the target cluster resource utilization rate is lower than the first utilization threshold for consecutive periods;

[0065] exponentially backoff the resource allocation frequency when the target cluster resource utilization rate is lower than the second utilization threshold.

[0066] In a second aspect, the embodiments of the present application provide a task scheduling system for big data, which is used for the task scheduling method for big data as described in the embodiments of the present application, and the system comprises:

[0067] a task scoring module, configured to obtain task parameters and dependency relationships between tasks, and calculate priority scores of the tasks;

[0068] a fitness obtaining module, configured to determine fitness of the tasks and the clusters according to data locality coefficients of the tasks, cluster parameters and cluster correlation correction coefficients;

[0069] a task scheduling module, configured to determine target clusters of the tasks and task orders according to the priority scores of the tasks and the fitness of the tasks and the clusters;

[0070] a fuse migration module, configured to trigger fuse when the number of task failures exceeds a threshold, migrate dependent tasks to the current cluster and recalculate the fitness if the cross-cluster dependency fails, and restore the priority of the task after consecutive successes.

[0071] In a third aspect, the present application further provides an electronic device, which comprises a memory, a processor and a computer program stored in the memory and capable of running on the memory, and the processor executes the program to implement the steps of any method of the present application.

[0072] The beneficial effects of the present application at least include: the task scheduling method for big data provided by the present application effectively solves the core problems of thread resource waste, priority strategy absence and middle station stability risk in the prior art by introducing task priority dynamic calculation, cluster fitness evaluation, fuse migration mechanism and resource balancing strategy; based on the multi-dimensional parameters such as synchronization time difference, failure times and dependency relationship of the task, the task priority is calculated through weighted average and correction coefficient, so that the high-value task can obtain resource allocation preferentially, and the invalid occupation of resources by the low-priority task is avoided; the task and the cluster are accurately matched by comprehensively considering the data locality coefficient, cluster network delay, CPU load and other factors, and the data cross-node transmission overhead is reduced; when the cluster resource utilization rate is lower than the threshold value, the linear proportional upgrade of the degraded queue task quota or the exponential backoff strategy is used to limit the allocation frequency, so that the resource allocation is more refined; when the task failure times exceed the threshold value, the fuse is automatically triggered, and for the cross-cluster dependency failure, the dependent task is migrated to the current cluster and the fitness is recalculated, so that invalid retry is avoided; for the task temporarily deployed in the suboptimal cluster, the resource state of the optimal cluster is continuously monitored, and when the condition is met, the secondary migration is triggered, and the migration priority weight is dynamically calculated, so that in the task migration, the multi-dimensional sorting such as the final priority score, the data locality coefficient and the arrival time is performed, and the pre-screening and resource pre-allocation mechanism are combined to ensure that the high-value task is executed preferentially. BRIEF DESCRIPTION OF DRAWINGS

[0073] Figure 1 is a task scheduling method for big data provided by an embodiment of the present application;

[0074] Figure 2 is a cluster allocation decision flowchart provided by an embodiment of the present application;

[0075] Figure 3 is a task scheduling flowchart provided by an embodiment of the present application;

[0076] Figure 4 is a task sorting flowchart provided by an embodiment of the present application. DETAILED DESCRIPTION

[0077] In the following, the present application will be further described in conjunction with the drawings and specific embodiments, and it should be noted that the following described embodiments or technical features can be combined in any manner to form new embodiments without conflict.

[0078] Referring to the drawings Figure 1 , referring to the drawings Figure 1 , an embodiment of the present application provides a task scheduling method for big data, and the method comprises:

[0079] obtaining task parameters and dependency relationship between tasks, and calculating task priority score;

[0080] The fitness of the task and the cluster is determined based on the task's data locality coefficient, cluster parameters, and cluster correlation correction coefficient.

[0081] The target cluster and task order are determined based on task priority scores and the fitness of tasks to clusters.

[0082] When the number of task failures exceeds the threshold, a circuit breaker is triggered. If the failure is due to cross-cluster dependency, the dependent task is migrated to the current cluster and the fitness is recalculated. The priority of the task is restored after consecutive successes.

[0083] The working principle and effects of the above technical solution are as follows: First, task parameters are acquired, then correction coefficients are determined based on inter-task dependencies, and finally, the final priority score of the task is obtained. This ensures that priority calculation considers not only the characteristics of the task itself but also the influence of dependencies. For example, for critical tasks upstream in the dependency chain, the system will increase their priority through correction coefficients to ensure their priority execution, thereby guaranteeing the efficient operation of the entire dependency chain. The basic fitness is calculated using data locality coefficients and cluster parameters, then priority weight coefficients are determined based on task priorities, and cluster correlation correction coefficients are considered to finally obtain the final fitness of the task and cluster. This multi-dimensional evaluation mechanism can accurately identify the most suitable cluster for task execution, reducing data transmission overhead and resource contention. Based on task priority scores and cluster fitness, task scheduling decisions are made to ensure that high-value tasks are executed first. When the number of task failures exceeds a threshold, a circuit breaker mechanism is triggered to suspend the execution of the task, avoiding unnecessary retries that consume resources. If the failure is due to cross-cluster dependency issues, the dependent tasks will be migrated to the current cluster, and fitness and priority will be recalculated. After continuous successful execution of a task, its original priority is gradually restored, achieving a smooth transition of task state. This mechanism can quickly respond to cluster anomalies and dependency failures, improving the system's fault tolerance and recovery speed.

[0084] In one possible implementation, the task parameters include the task synchronization time difference, synchronization success flag, number of task failures, synchronization period, synchronization time, and current task status; the task parameters and inter-task dependencies are obtained, and a task priority score is calculated, including:

[0085] The parameters of the task are normalized by interval mapping, and the initial priority score of the task is obtained by normalizing the parameters by weighted average.

[0086] The correction coefficient is determined based on the dependencies between tasks; the first priority score of the task is obtained based on the correction coefficient.

[0087]

[0088] in, score a first priority of the task; score an initial priority of the task; a correction coefficient, .

[0089] In a possible implementation, the parameters of the task are interval-mapped and normalized, and the initial priority of the task is scored by weighted average normalization of the parameters after processing; comprising:

[0090]

[0091] is the time interval between the nearest task synchronization operation to the current time and the preset reference time point; is the minimum value of the system task synchronization time difference in the statistical range; is the maximum value of the system task synchronization time difference in the statistical range; is the number of failures of the task; is the failure number blowout threshold; is the reference synchronization period; is the actual synchronization period of the task; is the maximum historical time consumption of the same type of task, that is, the longest time consumed by a single synchronization operation in the historical execution process of the task of the same type; is the actual synchronization time consumption of the task; is the current state coefficient of the task, 0.1 <0.8, such as 0.4 in the starting state, 0.6 in the running state, and 0.2 in the suspended state; 、 2、 3、 4、 5 is the first weight, .

[0092] In a possible implementation, the correction coefficient is determined according to the dependency relationship between tasks; comprising:

[0093]

[0094] wherein N is the number of direct downstream tasks of the current task; is the maximum number of downstream tasks in all tasks; is the dependency number influence factor, ∈[0.2,0.8], adjusted based on the feedback value (such as the downstream task success rate, delay rate, etc.) of the historical execution result.

[0095] The working principle of the above technical solution is: first, the synchronization time difference, synchronization success flag, failure number and other parameters of the task are obtained, and these parameters are normalized; for example, for the synchronization time difference parameter, it is mapped to the interval [0, 1], and the initial priority score of the task is calculated by weighted average, and then a correction coefficient is determined combined with the dependency relationship between tasks, and finally the final priority score of the task is obtained; this design makes the priority calculation not only consider the characteristics of the task itself, but also take into account the influence of the dependency relationship. For example, for a key task upstream of the dependency chain, the system will increase its priority through the correction coefficient to ensure its priority execution, thereby ensuring the efficient operation of the entire dependency chain; different weights reflect the different importance of each parameter on the priority of the task, such as a higher failure number weight, indicating that the failure number has a greater impact on the priority of the task. According to the dependency relationship between tasks, the correction coefficient is determined. The more the number of direct downstream tasks N of the task, the more critical it is in the dependency relationship, and the larger the correction coefficient; the dependency number influence factor is dynamically adjusted according to the historical execution result feedback value (such as the success rate of downstream tasks, delay rate, etc.) to adapt to different task dependency scenarios.

[0096] The beneficial effects of the above technical solution are: considering the task parameters and the dependency relationship between tasks, compared with determining the priority according to a single factor or fixed rules, the importance and execution order of the task in the entire task system can be more accurately reflected, and the scheduling rationality is improved; by determining the correction coefficient based on the number of downstream tasks and the dynamically adjusted dependency number influence factor, the task dependency structure can be adapted to various task dependency structures, whether it is a simple linear dependency or a complex mesh dependency relationship, the task priority can be reasonably adjusted to ensure the coherence and efficiency of task execution. The dependency number influence factor can be dynamically adjusted according to the historical execution result feedback, so that the priority calculation model can continuously learn and adapt to changes in the task execution environment, continuously optimize task priority evaluation, and improve the overall performance and stability of the system. Reasonable priority determination helps the scheduling system to allocate resources to high-priority tasks first, avoids wasting resources on low-value or non-critical tasks, improves resource utilization efficiency, and speeds up the overall task execution progress.

[0097] In one possible implementation, the cluster parameters include cluster network delay, cluster CPU load, and memory usage; the fitness of the task and the cluster is determined according to the data locality coefficient of the task, the cluster parameters, and the cluster correlation correction coefficient; including:

[0098] According to the locality coefficient of the task and the cluster parameters, the basic fitness of the task and the cluster is obtained;

[0099]

[0100] The benchmark delay is the cluster history average network delay, the benchmark CPU is the cluster theoretical maximum CPU utilization, and the benchmark memory is the cluster theoretical maximum memory usage; w1, w2, and w3 are second weight coefficients, w1 + =1; and are dynamically adjusted according to the task type (computation-intensive / IO-intensive);

[0101] The data locality coefficient is obtained in the following manner:

[0102]

[0103] The amount of data stored locally by the task on a cluster node; The total amount of data of the task; PL is the network path length from the computing node to the cluster data storage node when the task accesses data; PL is the shortest network path length from the computing node to each cluster data storage node when the task accesses data; The total number of times that the task accesses data of different cluster nodes within a preset time period; The total number of times that the task accesses data of the cluster node within a preset time period; x1, x2, and x3 are third weight coefficients, x1+x +x =1;

[0104] The priority weight coefficient is determined according to the task priority , and the final fitness of the task and the cluster is determined in combination with the cluster correlation correction coefficient :

[0105]

[0106] The cluster correlation correction coefficient is determined according to the inter-cluster network bandwidth, the inter-cluster data interaction frequency, and the historical task scheduling success rate.

[0107] The cluster correlation correction coefficient is determined according to the inter-cluster network bandwidth, the inter-cluster data interaction frequency, and the historical task scheduling success rate, including:

[0108] The inter-cluster network bandwidth, the inter-cluster data interaction frequency, and the historical task scheduling success rate are respectively normalized to [0, 1] according to the minimum value and the maximum value of the historical data, and then a weighted average is performed to obtain the cluster correlation correction coefficient;

[0109]

[0110] ​a1, a2, a3 are constants, a1>a2>a3, 1≤a1≤1.6, 0.8≤a2≤1.2, 0.5≤a3≤0.8; is the first priority score of the task after normalization mapping; for example, a1=1.5, a2=1, and a3=0.6.

[0111] The working principle of the above technical solution is as follows: first, the data locality coefficient is calculated, which comprehensively considers the close degree of the task data and the cluster from the proportion of the data stored locally by the task in the cluster node, the comparison of the network path length of accessing the data, and the proportion of the number of accessing the data in the cluster node to the total number of accessing the data, and obtains the data locality coefficient through weighted calculation; in combination with the cluster network delay, CPU load rate, memory usage, and the corresponding benchmark value (historical average network delay, theoretical maximum CPU utilization rate, and theoretical maximum memory usage), and dynamically adjusting the weight coefficient according to the task type, the basic fitness of the task and the cluster is obtained, which considers the influence of the local storage advantage of the task data and the current resource load of the cluster on the task execution; the priority weight coefficient and the cluster correlation correction coefficient are determined according to the range of the first priority score of the task (after normalization mapping), the priority weight coefficient is determined, different ranges correspond to different values, which reflects the influence of the task priority on the fitness of the task and the cluster. The network bandwidth between clusters, data interaction frequency, and historical task scheduling success rate are normalized and then weighted averaged to obtain the cluster correlation correction coefficient, which comprehensively considers the influence of the network connection condition between clusters, the data interaction activity, and the historical scheduling performance on the task scheduling; the basic fitness, the priority weight coefficient, and the cluster correlation correction coefficient are multiplied to obtain the final fitness of the task and the cluster, which comprehensively evaluates the fitness of the task and the cluster and provides a quantitative basis for scheduling the task to a suitable cluster.

[0112] The effect of the above technical solution is that the task data locality, cluster resource load, and cluster correlation characteristics are comprehensively considered, which can more accurately evaluate the fitness of the task and the cluster, schedule the task to the most suitable cluster for execution, reduce data transmission overhead, and improve resource utilization efficiency. According to the task type, the weight coefficient is dynamically adjusted, which can adapt to the characteristics of different types of tasks such as compute-intensive and IO-intensive, makes the scheduling strategy more flexible and targeted, and ensures that different types of tasks can be efficiently executed in a suitable cluster environment.

[0113] By considering dynamic change factors such as the network bandwidth between clusters and the data interaction frequency through the cluster correlation correction coefficient, and adjusting the weight coefficient according to the task priority, the adaptation evaluation of tasks to clusters can be dynamically adjusted according to environmental changes and the importance of tasks, continuously optimizing the task scheduling decision and improving the overall system performance and stability. The comprehensive and comprehensive adaptation evaluation and the cluster correlation correction based on historical data help to avoid scheduling tasks to unsuitable clusters, reduce the probability of task failure caused by improper cluster selection, and improve the task execution success rate.

[0114] Refer to the appendix Figure 2 , in a possible implementation manner, determining the target cluster and task sorting of the task according to the task priority score and the fitness of the task to the cluster; including:

[0115] If the priority score of a certain task is lower than the score threshold, and the final fitness of the task to a certain cluster is greater than b1 times the fitness of the task to the current cluster, then take this cluster as the target cluster of the task; b1 1.2, and determine the specific value according to the cluster load balancing requirement and the task execution efficiency requirement;

[0116] According to the task parameters and the locality gain coefficient of the target cluster, calculate the final priority score SC of the task in the target cluster new ;

[0117]

[0118] is the gain coefficient of the target cluster; ; is the first priority score of this task;

[0119] Among them, determine the target cluster gain coefficient according to the data localization ratio required by the task, the CPU utilization rate of the target cluster, and the memory usage rate of the target cluster;

[0120] Exemplarily, if the data localization ratio Local = 100% required by the task, and the CPU utilization rate Ucpu of the target cluster < Ucpu_low (Ucpu_low is the CPU utilization rate threshold of the target cluster, for example, Ucpu_low = 30%) and the memory usage rate Umem < Umem_low (Umem_low is the memory usage rate threshold of the target cluster, for example, Umem_low = 40%), then = 1.5; if 30% < Local < 100%, and 30% < Ucpu < 70% and 40% < Umem < 80%, then = 1.2; if L < 30%, or Ucpu > 70\%, Umem > 80\%, then = 0.8;

[0121] If the final task priority score of the task in the target cluster is greater than b2 times the first task priority score of the task in the original cluster, the task is moved to the target cluster.

[0122] If the final task priority score of the task in the target cluster is greater than b2 times the first task priority score of the task in the original cluster, the task is moved to the target cluster. 1.2; the specific value is determined according to the task migration cost and benefit evaluation.

[0123] The working principle of the technical solution is as follows:

[0124] Firstly, it is judged whether the task priority score is lower than the set score threshold, and the relationship between the task and the final fitness of a certain cluster and the fitness of the current cluster is compared (b1 times of the fitness of the current cluster is required); if the conditions are met, the cluster is taken as a target cluster, for example: the current cluster load (which can be calculated by CPU utilization, memory usage and other indicators) and the candidate cluster load can be obtained, the difference between the current cluster load and the candidate cluster load is calculated to determine the b1 value: according to the difference, b1 can be appropriately increased when the difference is large (indicating that the candidate cluster load is much lower than the current cluster), but the upper limit (such as 1.5) is limited to avoid excessive task migration; when the difference is small, b1 can take a value close to the lower limit 1.2; or, according to historical data and the current state of the cluster, the execution time of the task in the current cluster and the execution time of the task in the candidate cluster are estimated, if the execution time of the task in the candidate cluster < the execution time of the task in the current cluster, b1 can be determined according to the time difference value ratio, and a reasonable upper limit is set to ensure that the task can be migrated to a cluster with higher execution efficiency, while avoiding excessive frequent migration; this step comprehensively considers the task priority and the cluster fitness to select a cluster that may be more suitable for task execution. According to the task parameters and the target cluster locality gain coefficient, the final priority score of the task in the target cluster is calculated. The target cluster locality gain coefficient is determined according to the data localization ratio required by the task, the CPU utilization of the target cluster and the memory usage, which comprehensively considers the influence of the resource status of the target cluster and the data localization demand of the task on the task priority. Task migration decision: compare the final priority score of the task in the target cluster with the first priority score SC of the task in the original cluster, if it is greater than b2 times of SC, the task is migrated to the target cluster; the migration cost is large, b2 should take a larger value, the migration benefit is large, b2 can be appropriately reduced; when there are multiple target clusters that meet the conditions, the cluster with the highest final fitness of the task is preferentially selected for migration to ensure that the task can be executed in the most suitable cluster, and the task execution order is arranged according to the final priority score in the final target cluster task list.

[0125] The effect of the above technical solution is:

[0126] The target cluster is determined by comprehensively scoring the task priority and the cluster fitness, the task can be scheduled to a more suitable cluster, the resource waste in the mismatched task-cluster combination is avoided, and the overall utilization efficiency of the cluster resource is improved; the priority score in the target cluster is calculated according to the resource status of the target cluster and the task characteristics, so that the task can obtain the resource in the new cluster based on a more reasonable priority, the task waiting time is reduced, the task execution progress is accelerated, and the overall execution efficiency is improved; when multiple target clusters are available, the migration cluster is comprehensively decided according to the fitness and the priority score, and the tasks are reasonably sorted, so that the scheduling system can be flexibly adjusted according to different tasks and cluster conditions, and adapt to the complex and changeable task execution environment; through strict priority score comparison and migration decision rules, the high-priority task can be migrated to a more favorable cluster for execution, the high-priority task execution is prevented from being blocked due to cluster adaptation problems, and the smooth progress of the key task is ensured.

[0127] In a possible implementation, the method further includes:

[0128] When multiple tasks are migrated to the same cluster, the following rules are used for sorting:

[0129] All the migrated tasks are arranged in descending order according to the final priority score. If the final priority scores of the tasks are different, the tasks are arranged in descending order according to the final priority scores. If the final priority scores of the tasks are the same, the tasks are arranged in descending order according to the data locality coefficients of the tasks.

[0130] The resource capacity of the target cluster is checked before migration, and the migration is performed only when the migration condition is met.

[0131]

[0132] The task set to be migrated to the target cluster is denoted as T. The demand of the task i in the task set T for the resource is denoted as di. The safety margin coefficient is denoted as α, and the range is [0.1, 0.3]. The specific value is determined according to the cluster stability and the task fault tolerance requirement.

[0133] The working principle of the above technical solution is as follows:

[0134] When multiple tasks are to be migrated to the same cluster, the tasks are first arranged in descending order according to the final priority scores of the tasks in the target cluster, so as to preferentially guarantee the resource acquisition and execution order of the high-priority tasks. If the same, compare the data locality coefficient of the task, and the task with high data locality coefficient is preferred because high data locality means small data transmission overhead and can improve execution efficiency. If the data locality coefficient is still the same, arrange in ascending order according to the task arrival time, and the task that arrives first is processed first to ensure fairness. Migration condition judgment: before executing task migration, the system checks the target cluster resource capacity. By calculating the total amount of resource demand of each task in the task set to be migrated to the target cluster, and comparing it with the remaining resources of the target cluster (considering the safety margin coefficient), only when the total amount of resource demand of the task set does not exceed the amount of the remaining resources of the target cluster after deducting the safety margin, the task migration is allowed to prevent excessive migration from causing insufficient resources of the target cluster and affecting the stable operation of the cluster; among them, the fluctuation range of the cluster resource usage in the past period (such as the past week, month) can be counted (such as calculating the standard deviation of CPU, memory, etc. resource usage), if the fluctuation range is large, it means that the cluster resource usage fluctuates greatly, in order to ensure stability, a larger value should be taken, for example:

[0135]

[0136] the maximum fluctuation range allowed in the statistical range;

[0137] Alternatively, according to the task characteristics, the tasks are divided into high fault tolerance, medium fault tolerance, low fault tolerance, etc. levels, for example, batch processing tasks can be considered as high fault tolerance, and real-time transaction tasks are considered as low fault tolerance; when the proportion of low fault tolerance tasks in the migrated task set is large, in order to prevent insufficient resources from affecting the execution of these tasks, a larger value should be taken, for example: if the proportion of high fault tolerance tasks is large, a smaller value can be appropriately reduced.

[0138] The effect of the above technical solution is: by sorting according to the final priority score, it is ensured that high priority tasks can obtain resources and execute preferentially after migrating to the target cluster, avoiding low priority tasks from occupying resources, and ensuring the execution progress and efficiency of critical tasks; in the case of the same priority, the tasks with good data locality are arranged preferentially according to the data locality coefficient, reducing data transmission overhead and improving the overall resource utilization efficiency of the cluster. Maintain cluster stability: by setting a safety margin coefficient, a certain resource space is reserved to prevent the target cluster from running out of resources due to excessive task migration, maintain the stable operation of the cluster, and reduce the risk of task failure or cluster failure due to insufficient resources. Fair and orderly scheduling: arrange in ascending order according to the task arrival time to provide a fair scheduling order for tasks with the same priority and data locality, avoid task scheduling chaos, and ensure the orderliness of system scheduling.

[0139] ​Reference is made to the drawings Figure 3 - the drawings Figure 4 In a possible implementation, the method further includes:

[0140] Pre-screening the set of tasks to be migrated, and marking mandatory single-cluster tasks;

[0141] Specifically, the set of tasks to be migrated is pre-screened, and mandatory single-cluster tasks are marked, where a task satisfying any of the following conditions is a mandatory single-cluster task:

[0142] a data locality coefficient is greater than or equal to a coefficient threshold (for example, 0.9) and a target cluster data replica completeness is greater than or equal to a completeness threshold (for example, 95%);

[0143] there is an atomic service in a task dependency chain that must be bound to the current cluster;

[0144] an allow_migration=false flag is explicitly specified in a task parameter;

[0145] the mandatory single-cluster tasks are arranged in descending order of final priority scores, and an independent queue is generated;

[0146] tasks in the generated independent queue are preferentially submitted to their unique target cluster, and if a migration condition is met, resources are immediately allocated and locked; if not, a cross-cluster resource coordination protocol is triggered, and resources of tasks in a degraded queue are preferentially preempted; the cross-cluster resource coordination protocol specifically includes the following steps: first, the idle conditions of resources of other clusters are queried, and matching is performed according to the principle of similar resource requirements; if the matching is successful, resource allocation is performed; if the matching fails, tasks are queued or the task scheduling strategy is re-evaluated according to the task priority and the emergency degree of resource requirements.

[0147] The working principle of the above technical solution is as follows:

[0148] The set of tasks to be migrated is comprehensively scanned, and mandatory single-cluster tasks are identified and marked according to standards such as task attributes and business requirements. These tasks can only be executed in a specific cluster due to their own characteristics (such as data strong consistency requirements, specific deployment restrictions, etc.).

[0149] For the marked mandatory single-cluster tasks, they are arranged in descending order of final priority scores in the target cluster, and the tasks with high priority are arranged in the front row to form an independent queue. This arrangement ensures that high-value and key tasks can be processed preferentially.

[0150] The tasks in the independent queue are sequentially and preferentially submitted to their unique target cluster; before submission, it is checked whether the target cluster resource capacity meets the migration condition (such as the total resource demand being less than the amount of the remaining resources of the target cluster minus the safety margin); if it does, the task is immediately allocated resources and locked to prevent the resources from being preempted by other tasks, thereby ensuring the smooth execution of the task; if it does not, the cross-cluster resource coordination protocol is triggered.

[0151] Cross-cluster resource coordination: after triggering the protocol, the idle resources of other clusters are first queried. Matching is performed according to the principle of similar resource demand (such as similar demand for CPU, memory, storage, etc.); if matching idle resources can be found, resource allocation is performed to transfer resources from the idle cluster to the target cluster to meet the task demand; if matching fails, the task is arranged to wait for suitable resources in queue or the task scheduling strategy is re-evaluated (such as adjusting the task execution order, finding a new target cluster, etc.) according to the task priority and the urgency of resource demand.

[0152] The effect of the above technical solution is:

[0153] Through pre-screening and independent queue mechanism, forced single-cluster tasks are preferentially processed, ensuring that key tasks that can only be executed in a specific cluster due to business characteristics or technical limitations are processed in time, avoiding task delays or failures due to resource competition, and ensuring normal business operation. The cross-cluster resource coordination protocol can effectively utilize the idle resources of other clusters, avoid resource waste, and improve the resource utilization rate of the entire cluster system. At the same time, matching according to the principle of similar resource demand can make resource allocation more accurate and reasonable, reducing resource fragmentation. When the target cluster is short of resources, the cross-cluster resource coordination mechanism provides multiple coping strategies (queuing, re-evaluating the scheduling strategy, etc.), enabling the system to flexibly cope with resource shortages, enhancing the robustness and fault tolerance of the system in complex environments, and reducing the risk of system failure due to resource problems; forced single-cluster tasks are sorted by priority and generated into independent queues, which standardizes the task scheduling order, making task execution more planned and predictable, reducing task scheduling chaos and conflicts, and improving overall scheduling efficiency.

[0154] In a possible implementation manner, the method further includes:

[0155] If a task has at least one other target cluster in addition to the optimal target cluster that meets the migration condition, the task is marked as Ta multi ;

[0156] Initialize the candidate cluster list CL = [Cluster1, Cluster2, …, Clustern] in descending order of the final fitness of each cluster to the task;

[0157] Calculate the task Ta multithe difference between the fitness of the task and the optimal cluster and the maximum fitness of the task and other clusters;

[0158]

[0159] wherein, the fitness of the task and the optimal cluster; the maximum fitness of the task and other clusters;

[0160] in descending order, in the target cluster, the task with a large difference in fitness is preferentially processed;

[0161] determining whether the optimal cluster remaining resources can meet the current task; if yes, assigning the migration task to the optimal cluster, and locking the task in other candidate clusters for migration qualification; if no, trying the suboptimal cluster;

[0162] When the task is migrated to the suboptimal cluster, the resource state of the optimal cluster is continuously monitored; if the following conditions are met simultaneously:

[0163] the resource utilization rate of the optimal cluster is less than the utilization threshold (for example, 60%) for a plurality of consecutive periods (for example, 3 periods), the optimal cluster migration condition is met, the ratio of the final fitness of the task and the optimal cluster to the final fitness of the task and the current cluster is greater than b1*b3 times, and the duration of the task staying in the suboptimal cluster is greater than or equal to b4*the synchronization period; then triggering secondary migration, b3 and b4 are constants, b3>1.1; b4>2;

[0164] The secondary migration priority weight is set as:

[0165]

[0166] the secondary migration priority weight, the final priority score of the task in the current suboptimal cluster; the fitness of the task and the optimal target cluster; the fitness of the task and the current suboptimal cluster;

[0167] When the resource gap of a plurality of consecutive candidate clusters exceeds the preset proportion of the idle resources thereof, a cross-cluster resource coordination request is triggered.

[0168] The working principle of the above technical solution is:

[0169] When the task has other target clusters meeting the migration condition in addition to the optimal target cluster, the target cluster is marked as Ta multi ​; initialize a candidate cluster list CL, which includes all target clusters meeting the conditions, and arrange the clusters in descending order of final fitness of each cluster to the task, to provide an ordered cluster selection sequence for subsequent task allocation; calculate the task Ta multi The difference between the fitness of the optimal cluster (Cluster1) and the maximum fitness of other clusters, which measures the difference in the degree of adaptation between the optimal cluster and other better clusters. Then, arrange in descending order of the difference, and preferentially process the task with a large fitness difference, i.e., preferentially consider the task with a significant difference in the degree of adaptation between the optimal cluster and other clusters, to ensure that the task can be preferentially allocated to the most suitable cluster; determine whether the remaining resources of the optimal cluster can meet the current task demand. If yes, allocate the task to the optimal cluster, and lock the migration qualification of the task in other candidate clusters, to avoid resource waste and repeated scheduling; if not, try the suboptimal cluster, and check whether the resources meet the task demand in order according to the candidate cluster list. After the task is migrated to the suboptimal cluster, the resource state of the optimal cluster is continuously monitored. If a series of conditions are met (e.g., the resource utilization rate of the optimal cluster is lower than a threshold for a plurality of consecutive periods, the final fitness ratio of the task to the optimal cluster and the current cluster is greater than (b1*b3) times, and the task stays in the suboptimal cluster for a certain period of time), secondary migration is triggered. The priority weight of secondary migration is calculated according to the final priority score of the task in the current suboptimal cluster, the fitness of the optimal target cluster, and the fitness of the current suboptimal cluster, so that the task can be reasonably arranged in the migration order according to its adaptation and priority in different clusters during secondary migration. When the resource gap of a plurality of consecutive candidate clusters exceeds the preset proportion of the idle resources, it is indicated that the resource allocation of the current cluster is difficult, a cross-cluster resource coordination request is triggered, and the resources of other clusters are coordinated to meet the task demand, to ensure the smooth execution of the task.

[0170] The above technical solution has the following effects:

[0171] By calculating the fitness difference value and sorting, the task is preferentially assigned to the cluster with large fitness difference, so that the task can be executed in the most suitable cluster to the greatest extent, the problems of resource waste and low task execution efficiency caused by improper cluster selection are reduced, and the overall resource utilization efficiency is improved; in initial cluster allocation, the optimal cluster is preferentially considered, and if the resources are insufficient, the suboptimal cluster is tried, and after the task is migrated to the suboptimal cluster, the resource state of the optimal cluster is continuously monitored, and when the condition is met, secondary migration is triggered. This dynamic resource allocation strategy can more flexibly cope with changes in cluster resources, guarantee the resource supply of the task, and improve the rationality and effectiveness of system resource allocation; the secondary migration mechanism and the setting of the cross-cluster resource coordination request enable the system to flexibly adjust the task deployment according to the cluster resource state and the task adaptation, so that the system can respond in time when the cluster resources are tight or the task adaptation changes, and the stability and fault tolerance of the system in a complex and changeable environment are enhanced, and the risk of task failure due to resource problems is reduced; the calculation method of the secondary migration priority weight combines the fitness of the task in different clusters and the current priority, so that the high-value and high-priority task can be preferentially migrated to the optimal cluster for execution when the cluster resources allow, and the execution efficiency and effect of the key task are ensured.

[0172] In a possible implementation manner, the method further includes:

[0173] If the benefit ratio is detected for a plurality of times continuously < the benefit ratio threshold value (for example, 0.5), automatic rollback is triggered; for example, if the benefit ratio is detected for 3 times continuously < the benefit ratio threshold value, automatic rollback is triggered; wherein the benefit ratio = (task execution time after migration-task execution time before migration) / migration time consumption;

[0174] Global resource rebalancing is performed every k (for example, 2≤k≤5) scheduling periods, including:

[0175] Efficient migration caused by resource fragmentation is released;

[0176] The fitness difference value of all in-transit migration tasks is recalculated;

[0177] Active task recall is initiated for a cluster with a resource utilization rate lower than a threshold value;

[0178] Priority decay is applied to a task that stays in a suboptimal cluster for more than m periods (for example, 3≤m≤8):

[0179]

[0180] Wherein b5 is a constant, 0<b5<1; is the final priority score of the task currently in the suboptimal cluster; is the final priority score of the task initially in the suboptimal cluster;

[0181] When the priority after attenuation is lower than the migration threshold value, cross-cluster dependency reorganization is triggered, and the dependency relationship and scheduling strategy of the task and the cluster are reevaluated;

[0182] When the number of consecutive times of rejection of the migration request by the cluster reaches a threshold value g:

[0183] The duration of freezing the migration operation of the cluster is TC = min (2^g x base time, maximum freezing time);

[0184] For example, the base time can be 5 minutes, and the maximum freezing time can be 2 hours;

[0185] The working principle of the above technical solution is:

[0186] The benefit of task migration is evaluated by calculating the benefit ratio (task execution time after migration - task execution time before migration) / migration time consumption. If the benefit ratio is less than 0.5 for multiple times (such as 3 times), it means that the migration does not achieve the expected effect, and even may reduce the task execution efficiency. At this time, automatic rollback is triggered to restore the task to the state before migration, avoiding wasting resources on inefficient migration. Global resource rebalancing: global resource rebalancing is performed every k scheduling periods to identify and release resources occupied by inefficient migration tasks caused by resource fragmentation, and to re-integrate and utilize the resources. For all in-transit migration tasks, the fitness difference between the tasks and each cluster is recalculated, and the migration decision of the task is adjusted according to the cluster resource state and task demand changes. For clusters with resource utilization rate lower than a threshold value, active task recall is initiated to migrate part of the tasks from the cluster with low resource utilization rate to improve the resource utilization rate. For tasks that have been in the suboptimal cluster for more than m periods, priority attenuation is applied through a formula. With the passage of time, the priority of the task decreases. When the priority after attenuation is lower than the migration threshold value, cross-cluster dependency reorganization is triggered, and the dependency relationship and scheduling strategy of the task and the cluster are reevaluated, prompting the task to migrate to a more suitable cluster. Cluster freezing mechanism: when the number of consecutive times of rejection of the migration request by the cluster reaches a threshold value g, the migration operation of the cluster is frozen for a duration to prevent the cluster from causing resource scheduling chaos due to excessive rejection of requests. The base time (such as 5 minutes) and the maximum freezing time (such as 2 hours) are preset values to allow the cluster to have time to recover the resource state or adjust the scheduling strategy.

[0187] The effect of the above technical solution is:

[0188] The automatic rollback mechanism can correct inefficient migration in time, avoid wasting time and resources of tasks in inappropriate clusters, ensure that tasks are always executed in efficient environment, and improve the overall execution efficiency of tasks. The global resource rebalancing can make resources more reasonably distributed among clusters by processing inefficient migration, recalculating fitness difference, actively recalling tasks, etc., improve resource utilization, and avoid resource fragmentation and waste. Recalculating fitness difference regularly and decaying priority of tasks in suboptimal clusters and cross-cluster dependency reorganization can make the task scheduling system dynamically adjust according to the cluster resource state and task demand, better adapt to environmental changes, and guarantee the rationality and efficiency of task scheduling; and the cluster freezing mechanism avoids clusters from being in chaos due to continuous rejection of migration requests, gives clusters adjustment time through freezing operation, maintains the stability of clusters, and guarantees the reliable operation of the whole system.

[0189] In a possible implementation, the method further includes:

[0190] When the resource utilization rate of the target cluster is below a first utilization threshold (determined according to cluster resource planning and business demand) for consecutive multiple periods, the local resource quota of the tasks in the degradation queue is increased in a linear proportion, and the proportion is determined according to the resource idle condition and the task priority.

[0191] When the resource utilization rate of the target cluster is below a second utilization threshold (determined according to the minimum resource guarantee requirement of the cluster), an exponential backoff strategy is used to limit the resource allocation frequency, so as to avoid the performance degradation of the cluster caused by excessive resource allocation. The exponential backoff strategy is specifically: the backoff time interval is determined according to the difference between the resource utilization rate and the utilization threshold, and the resource allocation time interval is adjusted in an exponential growth manner.

[0192] The working principle and effects of the above technical solutions are as follows:

[0193] When the resource utilization (such as CPU, memory utilization) of the target cluster is continuously lower than the first utilization threshold (such as 40%, determined by cluster resource planning and business requirements) for multiple periods (such as 3 periods, the period length can be configured), it indicates that the cluster resources are in a lightly idle state. Identify the tasks currently in the degradation queue (low priority task queue), and according to the task priority (such as task priority score SC) and resource idle condition, linearly increase the local resource quota of the task. For example, if the resource utilization is lower than the threshold by 10%, the quota is increased by 5% / period, until it reaches 30% of the total cluster resources or the upper limit of the task priority quota. The calculation formula of the linear proportion is: new quota = linear coefficient x (first threshold - current utilization) x total cluster resources, wherein 0 < linear coefficient < 1, which is dynamically adjusted according to the task priority (higher priority tasks correspond to higher linear coefficients). Dynamic adjustment: after each increase in quota, the cluster utilization is re-evaluated, and if the utilization rises above the threshold, the increase is stopped; if it continues to be idle, the proportion is continued to be allocated until the pre-set upper limit is reached; the linear increase mechanism allows the lightly idle resources (above the second threshold and below the first threshold) to be reasonably utilized by the degraded queue tasks, avoiding resource waste.

[0194] When the target cluster resource utilization is lower than the second utilization threshold (such as 20%, determined by the minimum resource guarantee requirement of the cluster), it indicates that the cluster resources are in a serious idle or unstable state, and it is necessary to avoid sudden load impact caused by frequent resource allocation. Execution logic: the system starts an exponential backoff strategy, calculates the time interval (T) of resource allocation according to the difference between the current utilization and the second threshold, and the interval time increases exponentially with the number of failures (or the number of low utilization periods); by exponentially extending the allocation interval, the number of inefficient resource application operations is reduced, and the performance of the cluster is prevented from suddenly dropping under sudden load.

[0195] The embodiment of the application also provides a big data task scheduling system for the big data task scheduling method described in the embodiment of the application, and the system comprises:

[0196] A task scoring module is configured to obtain task parameters and inter-task dependency relationships, and calculate a task priority score.

[0197] An adaptability obtaining module is configured to determine the adaptability of a task to a cluster according to a data locality coefficient of the task, cluster parameters and a cluster correlation correction coefficient.

[0198] A task scheduling module is configured to determine a target cluster of a task and a task order according to a task priority score and the adaptability of the task to the cluster.

[0199] The fuse migration module is used for triggering fuse when the number of task failures exceeds the number threshold, if the cross-cluster dependency fails, migrating the dependent task to the current cluster and recalculating the fitness, and restoring the task priority after continuous success.

[0200] The principle and effect of the technical solution are the same as those of the method embodiments of the application, and will not be repeated here.

[0201] The application further provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the steps of the method according to any one of the application when executing the computer program.

[0202] The application is described from the viewpoints of use purpose, efficiency, progress and novelty, and meets the functional improvement and use requirements emphasized by the patent law. The above description and drawings are only preferred embodiments of the application, and do not limit the application. Therefore, all similar, identical, equivalent replacement or modification made within the scope of the patent application of the application shall belong to the protection scope of the patent application of the application.

Claims

1. A method for task scheduling of big data, characterized in that, The method comprises: obtaining task parameters and inter-task dependency relationship, and calculating task priority score; determining the fitness of the task and the cluster according to the data locality coefficient of the task, the cluster parameters and the cluster correlation correction coefficient; determining the target cluster of the task and the task order according to the task priority score and the fitness of the task and the cluster; triggering a fuse when the number of task failures exceeds the threshold, and if the cross-cluster dependency fails, migrating the dependent task to the current cluster and recalculating the fitness, and restoring the task priority after continuous success; The task parameters include the synchronization time difference of the task, the synchronization success flag, the number of task failures, the synchronization period, the synchronization time consumption and the current state of the task; obtaining the task parameters and the inter-task dependency relationship, and calculating the task priority score, comprising: interval mapping and normalization are performed on each parameter of the task, and the initial priority score of the task is obtained by weighted average normalization processing of each normalized parameter; determining the correction coefficient according to the inter-task dependency relationship; obtaining the first priority score of the task according to the correction coefficient and the initial priority score of the task; wherein the correction coefficient is determined according to the number of direct downstream tasks of the current task, the maximum number of downstream tasks in all tasks and the dependency number influence factor; The cluster parameters include cluster network delay, cluster CPU load and memory usage; the fitness of the task and the cluster is determined according to the data locality coefficient of the task, the cluster parameters and the cluster correlation correction coefficient; comprising: obtaining the basic fitness of the task and the cluster according to the locality coefficient of the task and the cluster parameters; ； w1, w2 and w3 are second weight coefficients, and w1+w2+w3=1; determining the final fitness of the task and the cluster according to the task priority and the cluster correlation correction coefficient; wherein the cluster correlation correction coefficient is determined according to the inter-cluster network bandwidth, the data interaction frequency between clusters and the historical task scheduling success rate; wherein the data locality coefficient is obtained by the following method: ； the amount of data stored locally at the cluster node for the task; the total amount of data for the task; PL is the network path length from the computing node to the cluster data storage node when the task accesses data; the shortest network path length from the computing node to each cluster data storage node when the task accesses data; the total number of times the task accesses data of different cluster nodes within a preset time period; the total number of times the task accesses data of the cluster node within a preset time period; x1, x2, x3 are third weight coefficients, x1+x2+x3=1. 2.The method of claim 1, wherein, obtaining the first priority score of the task according to the correction coefficient and the initial priority score of the task; comprising: ； wherein is a first priority score for the task; is an initial priority score for the task; is a correction factor, .

3. The method of claim 1, wherein, determining the target cluster of the task and the task order according to the task priority score and the fitness of the task and the cluster; comprising: If the priority score of a task is lower than the score threshold, and the final fitness of the task with a cluster is greater than b1 times the fitness of the task with the current cluster, the cluster is taken as the target cluster of the task; b1 1.2; According to the task parameters and the target cluster locality gain coefficient, the final priority score SC of the task in the target cluster is calculated new ; ； a target cluster gain coefficient; a first priority score of the task; wherein the target cluster gain coefficient is determined according to a data localization ratio required by the task, a CPU utilization of the target cluster, and a memory usage of the target cluster.​ if the final task priority score of the task in the target cluster is greater than b2 times the first task priority score of the task in the original cluster, the task is moved to the target cluster; If the final task priority score of the task in the multiple target clusters is greater than b2 times the first priority score of the task in the original cluster, the task is preferentially selected to migrate to the cluster with the highest final fitness; and sorted according to the final priority score of the task in the final target cluster task list; b2 1.

2.

4. The method of claim 3, wherein, The method further comprises: When multiple tasks are migrated to the same cluster, the following rules are used for sorting: Rank all migration tasks by final priority score in descending order; if the same, in descending order of the data locality coefficient of the task; if the data locality coefficient of the task is still the same, in ascending order of the task arrival time; checking the target cluster resource capacity before migration, and only performing migration when the migration condition is met, the migration condition being: ； a set of tasks to be migrated to the target cluster; a demand of resource for task i in the set of tasks; a safety margin coefficient, ranging from [0.1, 0.3].

5. The method of claim 4, wherein, The method further comprises: pre-screening the task set to be migrated, and marking the forced single-cluster task; Scoring mandatory single cluster tasks by final priority Descending order, generate independent queue; the task in the generated independent queue is preferentially submitted to its unique target cluster, and if the migration condition is met, the resources are immediately allocated and locked; if not, a cross-cluster resource coordination protocol is triggered, and the resources of the degraded queue task are preferentially preempted.

6. The method of claim 5, wherein, The method further comprises: If there is at least one other target cluster satisfying the migration condition in addition to the optimal target cluster for a task, mark the task as Ta multi ; initializing the candidate cluster list, and arranging the clusters in descending order of the final fitness of the clusters and the task; Computational task Ta multi fitness of the optimal cluster and the difference between the fitness of the task and the maximum fitness of the other clusters. ； wherein, is the fitness of the task to the optimal cluster; is the maximum fitness of the task to other clusters; In accordance with In the target cluster, the tasks with large fitness difference are processed preferentially in descending order. determining whether the optimal cluster remaining resources can satisfy the current task; if yes, assigning the migration task to the optimal cluster, and locking the task migration qualification in other candidate clusters; if no, trying sub-optimal clusters; when the task migrates to the sub-optimal cluster, continuously monitoring the resource state of the optimal cluster; if the following conditions are met simultaneously: the resource utilization rate of the optimal cluster is less than the utilization rate threshold for continuous multiple periods, the optimal cluster migration condition is met, and the ratio of the final fitness of the task and the optimal cluster to the final fitness of the task and the current cluster is greater than b1*b3 times, and the task stays in the sub-optimal cluster for a length of time greater than or equal to b4* synchronization period; triggering secondary migration, b3>1.1; b4>2; the secondary migration priority weight is set as: ； a secondary migration priority weight, a final priority score for the task in the current suboptimal cluster; a fitness of the task to the optimal target cluster; a fitness of the task to the current suboptimal cluster; when the resource gap of the continuous multiple candidate clusters exceeds the preset proportion of the idle resources, triggering a cross-cluster resource coordination request.

7. The method of claim 6, wherein, The method further comprises: performing global resource rebalancing every k scheduling period; for the task staying in the sub-optimal cluster for more than m periods, applying priority decay: ； where 0 < b5 < 1; the final priority score for the task initially in the suboptimal cluster; the final priority score for the task initially in the suboptimal cluster; when the decayed priority is lower than the migration threshold, triggering cross-cluster dependency reorganization, reevaluating the dependency relationship between the task and the cluster and the scheduling strategy. 8.The method of Claim 1, wherein The method further comprises: when the resource utilization rate of the target cluster is lower than the first utilization rate threshold for continuous multiple periods, linearly increasing the local resource quota of the degraded queue task; when the resource utilization rate of the target cluster is lower than the second utilization rate threshold, adopting an exponential backoff strategy to limit the resource allocation frequency.

9. A task scheduling system for big data, configured to implement the task scheduling method for big data according to claim 1, characterized in that, The system comprises: a task scoring module for obtaining task parameters and inter-task dependency relationships, and calculating task priority scores; a fitness obtaining module for determining the fitness of a task and a cluster according to the data locality coefficient of the task, the cluster parameters, and the cluster correlation correction coefficient; a task scheduling module for determining the target cluster of a task and the task order according to the task priority score and the fitness of the task and the cluster; a fuse migration module for triggering fuse when the number of task failures exceeds the number threshold, migrating the dependent task to the current cluster and recalculating the fitness if the cross-cluster dependency fails, and restoring the task priority after continuous success.

Images