Big data task scheduling method and system

Abstract

The invention provides a task scheduling method and system for big data, and relates to the technical field of big data, and the method comprises the steps: obtaining task parameters and a dependency relationship between tasks, and calculating a task priority score; according to the data locality coefficient of the task, the cluster parameter and the cluster relevance correction coefficient, determining the fitness of the task and the cluster; determining a target cluster of the tasks and a task sequence according to the task priority score and the fitness of the tasks and the cluster; when the number of failure times of the task exceeds a time threshold value, fusing is triggered, if cross-cluster dependence fails, the dependent task is migrated to the current cluster, the fitness is recalculated, and the priority of the task is recovered after continuous success; by means of dynamically adjusting task priorities, optimizing cluster resource allocation, establishing fusing and migration mechanisms and the like, the core problems of thread waste, priority disability, stability risk and the like in the traditional technology are solved, so that the service stability and the processing efficiency of a data intermediate station in a complex scene are improved.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a big data task scheduling method and system. Background Art

[0002] With the rapid development of big data technology, multi-cluster architectures (such as Hadoop and Kubernetes clusters) have become the core deployment model for data centers to process massive amounts of data due to their powerful scalability and flexibility. However, this architecture faces many challenges in task scheduling, and the limitations of existing technologies are becoming increasingly apparent, mainly in the following key aspects: In a multi-cluster environment, some tasks often fail to synchronize data due to external factors such as cluster anomalies, network fluctuations, and resource contention. Traditional scheduling mechanisms rely on fixed retry strategies, which can cause synchronization threads to be occupied for extended periods during multiple retries. This not only wastes thread resources but also significantly increases overall synchronization latency, leading to a backlog of subsequent tasks and severely impacting the real-time nature of data processing and system throughput. For example, when a cluster experiences a brief network outage, numerous tasks that rely on that cluster will be blocked due to frequent retries, preventing other normal tasks from obtaining resources and executing in a timely manner.

[0003] Traditional task retry strategies, mostly based on fixed time intervals (such as exponential backoff algorithms), provide only simple control over task execution frequency, while completely ignoring the historical behavioral characteristics of tasks (such as number of failures, synchronization time, and dependencies). This one-size-fits-all approach prevents the scheduling system from dynamically adjusting task priorities based on their actual status. For example, traditional strategies cannot allocate more resources or adjust the scheduling order for high-priority tasks that frequently retry due to cross-cluster dependency failures. Meanwhile, ineffective retries of low-priority tasks may preempt limited system resources, resulting in inefficient and inflexible scheduling.

[0004] In high-concurrency scenarios, the disordered retries of a large number of abnormal tasks can easily lead to serious problems such as thread pool exhaustion and resource deadlocks. When the thread pool is filled with retry requests for failed tasks, the data center will be unable to process new task requests, resulting in service unavailability and even system crashes. In addition, the failure of tasks with cross-cluster dependencies can trigger a chain reaction. Anomalies in one cluster can spread to multiple clusters through dependencies, further exacerbating system instability. For example, during a major promotion on an e-commerce platform, if order synchronization tasks are frequently retried due to a cluster failure, the entire data center can be paralyzed, affecting real-time statistics of transaction data and business decision-making.

[0005] The existing big data task scheduling technology lacks dynamic adaptability and intelligence in multi-cluster environments, and it is difficult to meet the requirements of the data middle platform for high stability and high reliability. Summary of the Invention

[0006] Based on the above problems, the task scheduling method and system for big data of the present invention solves the core problems of thread waste, priority disability and stability risks in traditional technologies by combining a scheduling method with multi-dimensional information such as task parameters, dependencies, cluster status, etc., by dynamically adjusting task priorities, optimizing cluster resource allocation, and establishing circuit breaking and migration mechanisms, thereby improving the service stability and processing efficiency of the data middle platform in complex scenarios.

[0007] The purpose of the present invention is achieved by the following technical solutions: In a first aspect, the present invention provides a method for scheduling tasks in big data, the method comprising: Obtain task parameters and dependencies between tasks, and calculate task priority scores; Determine the fitness of the task and the cluster based on the task's data locality coefficient, cluster parameters, and cluster correlation correction coefficient; Determine the target cluster and sort the tasks based on the task priority score and the fitness of the task with the cluster; When the number of task failures exceeds the threshold, a circuit breaker is triggered. If the failure is due to cross-cluster dependency, the dependent tasks are migrated to the current cluster and the fitness is recalculated. After continuous success, the priority of the task is restored.

[0008] Preferably, the task parameters include the task synchronization time difference, synchronization success flag, number of task failures, synchronization cycle, synchronization time and current task status; obtaining task parameters and dependencies between tasks, and calculating task priority scores, including: Perform interval mapping normalization on each parameter of the task, and obtain the initial priority score of the task by weighted averaging the normalized parameters; Determine a correction factor based on the dependency relationship between tasks; and obtain the first priority score of the task based on the correction factor;

[0009] in, Score the first priority for the task; Score the initial priority of the task; is the correction factor, .

[0010] Preferably, the cluster parameters include cluster network delay, cluster CPU load, and memory usage; the determination of the adaptability of the task and the cluster based on the data locality coefficient of the task, the cluster parameters, and the cluster correlation correction coefficient includes: According to the locality coefficient of the task and the cluster parameters, the basic fitness of the task and the cluster is obtained;

[0011] w1, w2, w3 are the second weight coefficients, w1+ + =1; Determine the priority weight coefficient based on the task priority , combined with the cluster correlation correction coefficient , determine the final fitness of the task and the cluster :

[0012] The cluster correlation correction coefficient is determined based on the inter-cluster network bandwidth, the inter-cluster data interaction frequency, and the historical task scheduling success rate.

[0013] Preferably, the step of determining the target cluster of the task and the task sorting according to the task priority score and the fitness of the task and the cluster includes: If the priority score of a task is lower than the score threshold, and the final fitness of the task with a cluster is greater than b1 times the fitness of the task with the current cluster, then the cluster will be used as the target cluster of the task; b1 1.2; Calculate the final priority score SC of the task in the target cluster based on the task parameters and the target cluster locality gain coefficient new ;

[0014] is the target cluster gain coefficient; ; Score the first priority of the task; determine the target cluster gain coefficient based on the data localization ratio required by the task, the target cluster CPU utilization, and the target cluster memory utilization; If the final task priority score of the task in the target cluster is greater than b2 times the first task priority score of the task in the original cluster, the task is moved to the target cluster; If the final task priority scores of the task in multiple target clusters are all greater than b2 times the first priority score of the task in the original cluster, the cluster with the highest final fitness between the task and the cluster will be selected for migration; and the tasks will be sorted according to their final priority scores in the task list of the final target cluster; b2 1.2.

[0015] Preferably, the method further comprises: When multiple tasks are migrated to the same cluster, they are sorted according to the following rules: Score all migration tasks according to their final priority Sort in descending order; if If the data locality coefficients of the tasks are the same, they are sorted in descending order. If the data locality coefficients of the tasks are still the same, they are sorted in ascending order by their arrival time. Before migration, the target cluster resource capacity is checked and migration is performed only if the migration conditions are met. The migration conditions are:

[0016] A collection of tasks to be migrated to the target cluster; is the resource demand of task i in the task set; is the safety margin factor, ranging from [0.1, 0.3].

[0017] Preferably, the method further comprises: Pre-screen the set of tasks to be migrated and mark them as forced single-cluster tasks; Score the forced single-cluster tasks according to their final priority Arrange in descending order to generate an independent queue; Tasks in the generated independent queue are submitted to their only target cluster first. If the migration conditions are met, resources are immediately allocated and locked. If not, the cross-cluster resource coordination protocol is triggered to prioritize the resources of tasks in the downgraded queue.

[0018] Preferably, the method further comprises: If a task has at least one other target cluster that meets the migration conditions in addition to the optimal target cluster, the task is marked as Ta multi ; Initialize the candidate cluster list and sort it in descending order according to the final fitness of each cluster and task; Computing task Ta multi The fitness difference with the optimal cluster and the maximum fitness of the task with other clusters;

[0019] in, is the fitness of the task with the optimal cluster; is the maximum fitness of the task with other clusters; according to Sort in descending order. In the target cluster, tasks with large fitness differences are prioritized. Determine whether the remaining resources of the optimal cluster can meet the current task. If so, assign the migration task to the optimal cluster and lock the task's migration eligibility in other candidate clusters. If not, try the next best cluster. After the task is migrated to the suboptimal cluster, the optimal cluster resource status is continuously monitored. If the following conditions are met at the same time: If the resource utilization of the optimal cluster is less than the utilization threshold for multiple consecutive cycles, the optimal cluster migration condition is met, the ratio of the final fitness of the task with the optimal cluster to the final fitness of the task with the current cluster is greater than b1*b3 times, and the task stays in the suboptimal cluster for ≥b4×synchronization period, then secondary migration is triggered, b3 ≥ 1.1; b4 ≥ 2; The secondary migration priority weight is set as:

[0020] is the secondary migration priority weight, Score the final priority of the task in the current suboptimal cluster; is the fitness of the task and the optimal target cluster; is the fitness of the task with the current suboptimal cluster; When the resource gaps of multiple consecutive candidate clusters exceed the preset ratio of their idle resources, a cross-cluster resource coordination request is triggered.

[0021] Preferably, the method further comprises: Perform global resource rebalancing every k scheduling cycles; For tasks that have been stuck in the suboptimal cluster for more than m cycles, priority decay is applied:

[0022] Among them, 0 <b5<1; Score the final priority of the task in the suboptimal cluster. Score the final priority of the task initially in the suboptimal cluster; When the priority after decay is lower than the migration threshold, cross-cluster dependency reorganization is triggered, and the dependency relationship between tasks and clusters and the scheduling strategy are re-evaluated.

[0023] Preferably, the method further comprises: When the target cluster resource utilization rate is lower than the first utilization threshold for multiple consecutive periods, the local resource quota of the degraded queue task is increased in a linear proportion; When the target cluster resource utilization is lower than the second utilization threshold, an exponential backoff strategy is used to limit the resource allocation frequency.

[0024] In a second aspect, an embodiment of the present invention provides a big data task scheduling system, which is used in the big data task scheduling method described in an embodiment of the present invention. The system includes: The task scoring module is used to obtain task parameters and dependencies between tasks, and calculate task priority scores; The fitness acquisition module is used to determine the fitness of the task and the cluster based on the data locality coefficient of the task, the cluster parameters and the cluster correlation correction coefficient; The task scheduling module is used to determine the target cluster of the task and the order of tasks based on the task priority score and the fitness of the task and the cluster; The circuit breaker migration module is used to trigger the circuit breaker when the number of task failures exceeds the threshold. If the failure is due to cross-cluster dependency, the dependent tasks are migrated to the current cluster and the fitness is recalculated. After continuous success, the priority of the task is restored.

[0025] In a third aspect, the present invention further provides an electronic device comprising a memory, a processor, and a computer program stored in and executable on the memory, wherein the processor executes the program to implement the steps of any method described in the present invention.

[0026] The beneficial effects of the present invention include at least: the task scheduling method for big data proposed by the present invention effectively solves the core problems of thread resource waste, lack of priority strategy and middle-end stability risk in the prior art by introducing dynamic calculation of task priority, cluster fitness evaluation, fuse migration mechanism and resource balancing strategy; based on multi-dimensional parameters such as task synchronization time difference, number of failures, dependency, etc., the task priority is calculated by weighted average and correction coefficient, so that high-value tasks can obtain resource allocation first and avoid low-priority tasks from occupying resources ineffectively; comprehensively considering factors such as data locality coefficient, cluster network delay, CPU load, etc., accurately matches tasks and clusters, and reduces the overhead of data transmission across nodes; When the cluster resource utilization rate is lower than the threshold, the quota of downgraded queue tasks is increased linearly or the allocation frequency is limited by exponential backoff strategy, making resource allocation more refined. When the number of task failures exceeds the threshold, the circuit breaker is automatically triggered. In the case of cross-cluster dependency failure, the dependent tasks are migrated to the current cluster and the fitness is recalculated to avoid invalid retries. For tasks temporarily deployed in the suboptimal cluster, the optimal cluster resource status is continuously monitored. When the conditions are met, a secondary migration is triggered. The migration priority weight is dynamically calculated. When migrating tasks, they are sorted according to multiple dimensions such as final priority score, data locality coefficient, arrival time, etc., combined with pre-screening and resource pre-allocation mechanisms to ensure that high-value tasks are executed first. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Schematic diagram of a big data task scheduling method provided by an embodiment of the present invention; Figure 2 This is a schematic diagram of the cluster allocation decision process provided by an embodiment of the present invention; Figure 3 This is a schematic diagram of the task scheduling process provided by an embodiment of the present invention; Figure 4This is a schematic diagram of the task sorting process provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that, under the premise of no conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0029] Refer to the attached Figure 1 , refer to the attached Figure 1 , an embodiment of the present invention provides a task scheduling method for big data, the method comprising: Obtain task parameters and dependencies between tasks, and calculate task priority scores; Determine the fitness of the task and the cluster based on the task's data locality coefficient, cluster parameters, and cluster correlation correction coefficient; Determine the target cluster and sort the tasks based on the task priority score and the fitness of the task with the cluster; When the number of task failures exceeds the threshold, a circuit breaker is triggered. If the failure is due to cross-cluster dependency, the dependent tasks are migrated to the current cluster and the fitness is recalculated. After continuous success, the priority of the task is restored.

[0030] The working principle and effectiveness of the above technical solution are as follows: first, task parameters are obtained, then a correction factor is determined based on inter-task dependencies, ultimately resulting in a final priority score for the task. This allows priority calculation to take into account not only the characteristics of the task itself but also the influence of dependencies. For example, for critical tasks upstream in the dependency chain, the system uses a correction factor to increase their priority, ensuring their execution first, thereby ensuring the efficient operation of the entire dependency chain. A base fitness is calculated using the data locality coefficient and cluster parameters, and then a priority weight coefficient is determined based on the task priority. Cluster-related correction factors are also considered to ultimately determine the final fitness of the task and cluster. This multi-dimensional evaluation mechanism accurately identifies the cluster most suitable for task execution, reducing data transmission overhead and resource contention. Task scheduling decisions are made based on the task priority score and cluster fitness, ensuring that high-value tasks are prioritized. When the number of task failures exceeds a threshold, a circuit breaker mechanism is triggered, pausing the task's execution to avoid wasted resources due to ineffective retries. If the failure is caused by a cross-cluster dependency issue, the dependent task is migrated to the current cluster and its fitness and priority are recalculated. After a task successfully executes continuously, its original priority is gradually restored, achieving a smooth transition of task status. This mechanism can quickly respond to cluster anomalies and dependency failures, improving the system's fault tolerance and recovery speed.

[0031] In one possible implementation, the task parameters include the task synchronization time difference, synchronization success flag, number of task failures, synchronization cycle, synchronization time, and current task status; obtaining the task parameters and dependencies between tasks, and calculating the task priority score, including: Perform interval mapping normalization on each parameter of the task, and obtain the initial priority score of the task by weighted averaging the normalized parameters; Determine a correction factor based on the dependency relationship between tasks; and obtain the first priority score of the task based on the correction factor;

[0032] in, Score the first priority for the task; Score the initial priority of the task; is the correction factor, .

[0033] In a possible implementation, performing interval mapping normalization on each parameter of the task and obtaining an initial priority score of the task by weighted averaging the normalized parameters may include:

[0034] The time difference between the most recent synchronization of the task, that is, the time interval between the most recent task synchronization operation from the current time and the preset reference time point; The minimum value of the synchronization time difference of multiple tasks in the system within the statistical range; The maximum value of the synchronization time difference among multiple tasks in the system within the statistical range; The number of times the task failed; The failure number circuit breaker threshold; is the benchmark synchronization period; The actual synchronization period for the task; The maximum time spent in history for tasks of the same type, that is, the longest time spent on a single synchronization operation during the historical execution of tasks of the same category as this task; The actual synchronization time of the task; is the current state coefficient of the task, 0.1< <0.8, such as starting = 0.4, running = 0.6, paused = 0.2; 、 2. 3. 4. 5 is the first weight, .

[0035] In a possible implementation, determining the correction coefficient according to the inter-task dependency relationship includes:

[0036] Where N is the number of direct downstream tasks of the current task; is the maximum number of downstream tasks among all tasks; is the quantity-dependent impact factor, ∈[0.2,0.8], and is adjusted based on the feedback value of historical execution results (such as downstream task success rate, delay rate, etc.).

[0037] The working principle of the above technical solution is as follows: first, task parameters such as synchronization time difference, synchronization success flag, and number of failures are obtained and normalized. For example, the synchronization time difference parameter is mapped to the interval [0, 1]. An initial task priority score is calculated using a weighted average. Then, a correction factor is determined based on the inter-task dependencies to obtain the task's final priority score. This design allows the priority calculation to take into account not only the characteristics of the task itself but also the influence of dependencies. For example, for critical tasks upstream in the dependency chain, the system increases their priority using a correction factor to ensure priority execution, thereby ensuring the efficient operation of the entire dependency chain. Different weights reflect the varying importance of each parameter to the task's priority. For example, a higher weight for the number of failures indicates a greater impact on the task's priority. The correction factor is determined based on the inter-task dependencies. The greater the number of direct downstream tasks N, the more critical the task is in the dependency relationship, and the larger the correction factor. The dependency number influencing factor is dynamically adjusted based on historical execution feedback (such as the success rate and delay rate of downstream tasks) to adapt to different task dependency scenarios.

[0038] The beneficial effects of the above technical solution are: by comprehensively considering the task's own parameters and the dependencies between tasks, compared with determining the priority based on a single factor or fixed rules, it can more accurately reflect the importance of the task in the entire task system and the execution order, thereby improving the rationality of scheduling; by determining the correction coefficient based on the number of downstream tasks and the dynamically adjusted dependency quantity influencing factor, it can adapt to a variety of task dependency structures, whether it is a simple linear dependency or a complex mesh dependency relationship, and can reasonably adjust the task priority to ensure the consistency and efficiency of task execution. The dependency quantity influencing factor can be dynamically adjusted based on historical execution result feedback, so that the priority calculation model can continuously learn and adapt to changes in the task execution environment, continuously optimize task priority evaluation, and improve the overall performance and stability of the system. Reasonable priority determination helps the scheduling system to prioritize resources to high-priority tasks, avoid wasting resources on low-value or non-critical tasks, improve resource utilization efficiency, and accelerate the overall task execution progress.

[0039] In one possible implementation, the cluster parameters include cluster network latency, cluster CPU load, and memory usage; and determining the adaptability of the task and the cluster based on the task's data locality coefficient, cluster parameters, and cluster affinity correction coefficient includes: According to the locality coefficient of the task and the cluster parameters, the basic fitness of the task and the cluster is obtained;

[0040] The benchmark delay is the historical average network delay of the cluster, the benchmark CPU is the theoretical maximum CPU utilization of the cluster, and the benchmark memory is the theoretical maximum memory utilization of the cluster; w1, w2, and w3 are the second weight coefficients, w1+ + =1; and dynamically adjusted according to the task type (computation-intensive / IO-intensive); The data locality coefficient is obtained as follows:

[0041] The amount of data stored locally on a cluster node for this task; is the total data volume of the task; PL is the network path length from the computing node to the data storage node of the cluster when the task accesses data; PL is the shortest network path length from the computing node to each cluster data storage node when the task accesses data; The total number of times the task accesses data from different cluster nodes within the preset time period; is the total number of times the task accesses the cluster node data within the preset time period; x1, x2, and x3 are the third weight coefficients, x1+x +x =1; Determine the priority weight coefficient based on the task priority , combined with the cluster correlation correction coefficient , determine the final fitness of the task and the cluster :

[0042] The cluster correlation correction coefficient is determined based on the inter-cluster network bandwidth, the inter-cluster data interaction frequency, and the historical task scheduling success rate.

[0043] Determine the cluster affinity correction coefficient based on the inter-cluster network bandwidth, inter-cluster data exchange frequency, and historical task scheduling success rate, including: Based on the minimum / maximum values of historical data, the inter-cluster network bandwidth, inter-cluster data interaction frequency, and historical task scheduling success rate are normalized to [0, 1] and then weighted averaged to obtain the cluster correlation correction coefficient.

[0044] a1, a2, a3 are constants, a1>a2>a3, 1≤a1≤1.6, 0.8≤a2≤1.2, 0.5≤a3≤0.8; Score the first priority of the task after normalization mapping to [0,1]; for example, a1=1.5, a2=1, a3=0.6.

[0045] The working principle of the above technical solution is as follows: first calculate the data locality coefficient, and comprehensively consider the degree of association between the task data and the cluster from aspects such as the proportion of data stored locally in the cluster node, the comparison of the network path length for accessing the data, and the proportion of the number of data accesses on the cluster node to the total number of accesses, and obtain the data locality coefficient through weighted calculation; combine cluster parameters such as cluster network latency, CPU load rate, memory usage, and corresponding benchmark values (historical average network latency, theoretical maximum CPU utilization, theoretical maximum memory usage), and dynamically adjust the weight coefficient according to the task type to obtain the basic fitness of the task and the cluster. This process takes into account the local storage advantages of the task data and the impact of the current resource load of the cluster on the execution of the task; determine the priority weight coefficient and the cluster correlation correction coefficient. The priority weight coefficient is determined according to the range of the task's first priority score (after normalized mapping). Different ranges correspond to different values, reflecting the impact of the task priority on its fitness with the cluster. The inter-cluster network bandwidth, data interaction frequency, and historical task scheduling success rate are normalized and then weighted averaged to obtain the cluster correlation correction coefficient, which comprehensively considers the impact of inter-cluster network connection status, data interaction activity, and historical scheduling performance on task scheduling; the basic fitness, priority weight coefficient, and cluster correlation correction coefficient are multiplied together to obtain the final fitness of the task and the cluster, comprehensively and comprehensively evaluate the adaptability of the task and the cluster, and provide a quantitative basis for scheduling tasks to appropriate clusters.

[0046] The benefits of this technical solution include: By comprehensively considering multiple factors, including task data locality, cluster resource load, and inter-cluster correlations, it can more accurately assess task compatibility with the cluster and schedule tasks to the most suitable cluster for execution, reducing data transmission overhead and improving resource utilization. Dynamically adjusting the weight coefficient based on task type can adapt to the characteristics of different types of tasks, such as compute-intensive and I / O-intensive tasks, making scheduling strategies more flexible and targeted, ensuring that different types of tasks can be efficiently executed in the appropriate cluster environment.

[0047] By considering dynamic change factors such as the network bandwidth between clusters and the data interaction frequency through the cluster correlation correction coefficient, and adjusting the weight coefficient according to the task priority, the adaptation evaluation of tasks to clusters can be dynamically adjusted according to environmental changes and the importance of tasks, continuously optimizing the task scheduling decision-making and enhancing the overall system performance and stability. The comprehensive and comprehensive adaptation evaluation and the cluster correlation correction based on historical data help to avoid scheduling tasks to unsuitable clusters, reduce the probability of task failure caused by improper cluster selection, and improve the task execution success rate.

[0048] Refer to the appendix Figure 2 , in a possible implementation manner, determining the target cluster and task sorting of a task according to the task priority score and the fitness of the task to the cluster; includes: If the priority score of a certain task is lower than the score threshold, and the final fitness of the task to a certain cluster is greater than b1 times the fitness of the task to the current cluster, then this cluster is used as the target cluster of the task; b1 1.2, the specific value is determined according to the cluster load balancing requirement and the task execution efficiency requirement; According to the task parameters and the target cluster locality gain coefficient, calculate the final priority score SC of this task in the target cluster new ;

[0049] is the target cluster gain coefficient; ; is the first priority score of this task; Among them, the target cluster gain coefficient is determined according to the data localization ratio required by the task, the CPU utilization rate of the target cluster, and the memory usage rate of the target cluster; Exemplarily, if the data localization ratio Local = 100% required by the task, and the CPU utilization rate Ucpu of the target cluster < Ucpu_low (Ucpu_low is the CPU utilization rate threshold of the target cluster, for example, Ucpu_low = 30%), and the memory usage rate Umem < Umem_low (Umem_low is the memory usage rate threshold of the target cluster, for example, Umem_low = 40%), then = 1.5; if 30% < Local < 100%, and 30% < Ucpu < 70% and 40% < Umem < 80%, then = 1.2; if L < 30%, or Ucpu > 70\%, Umem > 80\%, then = 0.8; If the final task priority score of the task in the target cluster is greater than b2 times the first task priority score of the task in the original cluster, the task is moved to the target cluster; If the final task priority scores of the task in multiple target clusters are all greater than b2 times the first priority score of the task in the original cluster, the cluster with the highest final fitness between the task and the cluster will be selected for migration; and the tasks will be sorted according to their final priority scores in the task list of the final target cluster; b2 1.2; Determine the specific value based on the cost and benefit evaluation of task migration.

[0050] The working principle of the above technical solution is: First, determine whether the task priority score is lower than the set score threshold, and compare the relationship between the task and the final fitness of a cluster and the fitness of the current cluster (which must be greater than b1 times the fitness of the current cluster); if the conditions are met, then the cluster is used as the target cluster. For example, you can obtain the current cluster load (which can be calculated by weighting indicators such as CPU utilization and memory usage), the candidate cluster load, and calculate the difference between the current cluster load and the candidate cluster load to determine the b1 value: set b1 according to the difference. When the difference is large (indicating that the candidate cluster load is much lower than the current cluster), b1 can be appropriately increased, but An upper limit (e.g., 1.5) should be set to avoid excessive task migration. When the difference is small, b1 can be taken close to the lower limit of 1.2. Alternatively, based on historical data and the current cluster status, estimate the task's execution time in the current cluster and the candidate cluster. If the task's execution time in the candidate cluster is less than the task's execution time in the current cluster, b1 can be determined based on the time difference ratio. Similarly, a reasonable upper limit can be set to ensure that the task can be migrated to a cluster with higher execution efficiency while avoiding excessive migration. This step comprehensively considers both task priority and cluster adaptability to select clusters that may be more suitable for task execution. The final priority score of the task in the target cluster is calculated based on the task parameters and the target cluster's locality gain coefficient. The target cluster's locality gain coefficient is determined based on the task's required data locality ratio, the target cluster's CPU utilization, and memory usage, comprehensively considering the impact of the target cluster's resource availability and the task's data locality requirements on task priority. Task migration decision: Compare the final priority score of the task in the target cluster with the first priority score SC in the original cluster. If it is greater than b2 times SC, the task is moved to the target cluster. If the migration cost is large, b2 should be large; if the migration benefit is large, b2 can be appropriately reduced. When there are multiple target clusters that meet the conditions, the cluster with the highest final fitness between the task and the cluster is preferentially selected for migration to ensure that the task can be executed in the most suitable cluster. The task execution order is reasonably arranged according to the final priority score in the task list of the final target cluster.

[0051] The effects of the above technical solution are: By comprehensively determining the target cluster based on task priority scores and cluster fitness, tasks can be scheduled to more suitable clusters, avoiding resource waste on mismatched task-cluster combinations and improving the overall utilization efficiency of cluster resources. Priority scores in the target cluster are calculated based on the resource status of the target cluster and task characteristics, so that tasks can obtain resources in the new cluster based on more reasonable priorities, reducing task waiting time, speeding up task execution progress, and improving overall execution efficiency. When multiple target clusters are available, cluster migration is decided based on fitness and priority scores, and tasks are sorted reasonably, so that the scheduling system can flexibly adjust according to different tasks and cluster conditions and adapt to complex and changing task execution environments. Through strict priority score comparison and migration decision rules, high-priority tasks can be migrated to more favorable clusters for execution, avoiding the obstruction of high-priority task execution due to cluster adaptation issues, and ensuring the smooth progress of key tasks.

[0052] In a possible implementation, the method further includes: When multiple tasks are migrated to the same cluster, they are sorted according to the following rules: Score all migration tasks according to their final priority Sort in descending order; if If the data locality coefficients of the tasks are the same, they are sorted in descending order. If the data locality coefficients of the tasks are still the same, they are sorted in ascending order by their arrival time. Before migration, the target cluster resource capacity is checked and migration is performed only if the migration conditions are met. The migration conditions are:

[0053] A collection of tasks to be migrated to the target cluster; is the resource demand of task i in the task set; is the safety margin coefficient, ranging from [0.1, 0.3]; it is determined based on the cluster stability and task fault tolerance requirements Specific value.

[0054] The working principle of the above technical solution is: When multiple tasks are to be migrated to the same cluster, they are first sorted in descending order based on their final priority scores in the target cluster, giving priority to resource acquisition and execution order for high-priority tasks; if If they are the same, the data locality coefficients of the tasks will be compared, and the tasks with higher data locality coefficients will be given priority, because high data locality means less data transmission overhead, which can improve execution efficiency. If the data locality coefficients are still the same, the tasks will be sorted in ascending order by arrival time, and the tasks that arrive first will be processed first to ensure fairness. Migration condition judgment: Before executing task migration, the system will check the resource capacity of the target cluster. By calculating the sum of the resource requirements of each task in the task set to be migrated to the target cluster, and comparing it with the remaining resources of the target cluster (taking into account the safety margin coefficient); task migration is only allowed when the sum of the resource requirements of the task set does not exceed the amount of the remaining resources of the target cluster after deducting the safety margin, to prevent insufficient resources of the target cluster due to excessive migration, which will affect the stable operation of the cluster; among them, the fluctuation range of the cluster resource usage of the target cluster in the past period of time (such as the past week or month) can be counted. (Such as calculating the standard deviation of CPU, memory and other resource usage). If the fluctuation is large, it means that the cluster resource usage fluctuates greatly. To ensure stability, A larger value should be taken, for example:

[0055] The maximum fluctuation allowed within the statistical range; Alternatively, tasks can be divided into high fault tolerance, medium fault tolerance, and low fault tolerance levels based on their characteristics. For example, batch processing tasks can be considered high fault tolerance, while real-time trading tasks can be considered low fault tolerance. When low fault tolerance tasks account for a large proportion of the migration task set, in order to prevent insufficient resources from affecting the execution of these tasks, A larger value should be taken, for example: ; If the proportion of high fault tolerance tasks is large, Can be appropriately lowered.

[0056] The effects of the above technical solution are: by sorting by the final priority score, it is ensured that high-priority tasks can obtain resources and be executed first after migrating to the target cluster, avoiding low-priority tasks from preempting resources, and ensuring the execution progress and efficiency of key tasks; in the case of the same priority, sorting according to the data locality coefficient, giving priority to tasks with good data locality, reducing data transmission overhead, and improving the overall resource utilization efficiency of the cluster. Maintaining cluster stability: by setting a safety margin coefficient, a certain amount of resource space is reserved to prevent the exhaustion of target cluster resources due to excessive task migration, maintain the stable operation of the cluster, and reduce the risk of task failure or cluster failure due to insufficient resources. Fair and orderly scheduling: Arrange tasks in ascending order of arrival time, provide a fair scheduling order for tasks with the same priority and data locality, avoid task scheduling chaos, and ensure the orderliness of system scheduling.

[0057] Refer to the attached Figure 3 -Attached Figure 4 In a possible implementation, the method further includes: Pre-screen the set of tasks to be migrated and mark them as forced single-cluster tasks; Specifically, the set of tasks to be migrated can be pre-screened and marked as forced single-cluster tasks. Tasks that meet any of the following conditions are considered forced single-cluster tasks: The data locality coefficient is greater than or equal to the coefficient threshold (for example, 0.9) and the target cluster data replica integrity is greater than or equal to the integrity threshold (for example, 95%). There are atomic services in the task dependency chain that must be bound to the current cluster; The allow_migration=false flag is explicitly specified in the task parameters; Arrange the forced single-cluster tasks in descending order of their final priority scores to generate independent queues; The tasks in the generated independent queue will be submitted to their only target cluster first. If the migration conditions are met, the resources will be allocated and locked immediately. If not, the cross-cluster resource coordination protocol will be triggered to prioritize the resources of the downgraded queue tasks. The specific process of the cross-cluster resource coordination protocol is as follows: first query the resource idleness of other clusters, match them according to the principle of similar resource requirements, and if the match is successful, resource allocation will be carried out. If it fails, the task will be queued or the task scheduling strategy will be re-evaluated based on the task priority and the urgency of the resource demand.

[0058] The working principle of the above technical solution is: A comprehensive scan of the set of tasks to be migrated is performed. Based on criteria such as task attributes and business requirements, mandatory single-cluster tasks are identified and marked. These tasks can only be executed in a specific cluster due to their inherent characteristics (such as strong data consistency requirements and specific deployment restrictions).

[0059] For the marked forced single-cluster tasks, they are sorted in descending order according to their final priority scores in the target cluster. Tasks with high priority are placed at the front to form an independent queue. This sorting method ensures that high-value and critical tasks are processed first.

[0060] Tasks in independent queues are submitted to their unique target cluster in priority order. Before submission, the resource capacity of the target cluster is checked to see if it meets the migration conditions (for example, the total resource demand does not exceed the remaining resources of the target cluster after deducting the safety margin). If so, resources are immediately allocated to the task and locked to prevent them from being preempted by other tasks and ensure smooth task execution. If not, the cross-cluster resource coordination protocol is triggered.

[0061] Cross-cluster resource coordination: After the protocol is triggered, the system first queries the idle resources of other clusters. It then matches tasks based on similar resource requirements (e.g., tasks with similar CPU, memory, storage, and other resource requirements). If matching idle resources are found, resource allocation is performed, transferring resources from the idle cluster to the target cluster to meet the task's needs. If a match fails, the system queues the task for suitable resources based on its priority and urgency of resource needs, or re-evaluates the task scheduling strategy (e.g., adjusting the task execution order, searching for a new target cluster, etc.).

[0062] The effects of the above technical solution are: Through pre-screening and an independent queue mechanism, mandatory single-cluster tasks are prioritized, ensuring that critical tasks that can only be executed in a specific cluster due to business characteristics or technical limitations are promptly processed, avoiding task delays or failures due to resource competition and ensuring normal business operations. A cross-cluster resource coordination protocol effectively utilizes idle resources from other clusters, avoiding resource waste and improving resource utilization across the entire cluster system. Furthermore, matching tasks based on similar resource requirements ensures more precise and reasonable resource allocation and reduces resource fragmentation. When resources in the target cluster are insufficient, the cross-cluster resource coordination mechanism provides multiple response strategies (such as queuing and re-evaluating scheduling strategies), enabling the system to flexibly cope with resource shortages, enhancing its robustness and fault tolerance in complex environments, and reducing the risk of system failures caused by resource issues. Prioritizing mandatory single-cluster tasks and creating independent queues standardizes task scheduling, making task execution more planned and predictable, reducing scheduling confusion and conflicts, and improving overall scheduling efficiency.

[0063] In a possible implementation, the method further includes: If a task has at least one other target cluster that meets the migration conditions in addition to the optimal target cluster, the task is marked as Ta multi ; Initialize the candidate cluster list CL=[Cluster1, Cluster2, ..., Clustern] and sort them in descending order according to the final fitness of each cluster and task; Computing task Ta multi The fitness difference with the optimal cluster and the maximum fitness of the task with other clusters;

[0064] in, is the fitness of the task with the optimal cluster; is the maximum fitness of the task with other clusters; according to Sort in descending order. In the target cluster, tasks with large fitness differences are prioritized. Determine whether the remaining resources of the optimal cluster can meet the current task. If so, assign the migration task to the optimal cluster and lock the task's migration eligibility to other candidate clusters. If not, try the next best cluster. After the task is migrated to the suboptimal cluster, the optimal cluster resource status is continuously monitored. If the following conditions are met at the same time: Secondary migration is triggered if the resource utilization of the optimal cluster is less than a utilization threshold (e.g., 60%) for multiple consecutive cycles (e.g., 3 cycles), the optimal cluster migration condition is met, the ratio of the task's final fitness with the optimal cluster to the task's final fitness with the current cluster is greater than b1*b3 times, and the task's retention time in the suboptimal cluster is ≥b4×synchronization period. b3 and b4 are constants, b3 ≥ 1.1; b4 ≥ 2. The secondary migration priority weight is set as:

[0065] is the secondary migration priority weight, Score the final priority of the task in the current suboptimal cluster; is the fitness of the task and the optimal target cluster; is the fitness of the task with the current suboptimal cluster; When the resource gaps of multiple consecutive candidate clusters exceed the preset ratio of their idle resources, a cross-cluster resource coordination request is triggered.

[0066] The working principle of the above technical solution is: When there are other target clusters that meet the migration conditions besides the optimal target cluster, they are marked as Ta multi Initialize the candidate cluster list CL, include all target clusters that meet the conditions, and arrange them in descending order according to the final fitness of each cluster and task, providing an orderly cluster selection sequence for subsequent task allocation; calculate the task Ta multiThe difference between the task's fitness with the optimal cluster (Cluster 1) and its maximum fitness with other clusters is used to measure the difference in fitness between the optimal cluster and other less-optimal clusters. Next, tasks are sorted in descending order by fitness difference, prioritizing tasks with large fitness differences. Specifically, tasks with significant fitness differences between the optimal cluster and other clusters are prioritized to ensure that they are assigned to the most suitable cluster. The remaining resources in the optimal cluster are then determined to meet the current task's requirements. If so, the task is assigned to the optimal cluster and its migration eligibility in other candidate clusters is locked to avoid resource waste and duplicate scheduling. If not, the task is then relocated to the suboptimal cluster, checking its resources in the order listed in the candidate cluster list to see if they meet the task's requirements. After the task is migrated to the suboptimal cluster, the resource status of the optimal cluster is continuously monitored. A second migration is triggered if a series of conditions are met (e.g., the optimal cluster's resource utilization is below a threshold for multiple consecutive cycles, the optimal cluster migration conditions are met, the task's final fitness ratio with the optimal cluster and the current cluster is greater than (b1 * b3) times, or the task's retention time in the suboptimal cluster exceeds a certain threshold). The secondary migration priority weight is calculated based on the task's final priority score in the current suboptimal cluster, its fitness with the optimal target cluster, and its fitness with the current suboptimal cluster. This allows tasks to be properly ordered during secondary migration based on their adaptability and priority across clusters. When the resource gaps of multiple consecutive candidate clusters exceed the preset ratio of their idle resources, resource allocation in the current cluster becomes difficult, triggering a cross-cluster resource coordination request. This ensures smooth task execution by coordinating resources from other clusters to meet task requirements.

[0067] The effects of the above technical solution are: By calculating the fitness difference and sorting them, tasks are assigned to clusters with large fitness differences first, so that tasks can be executed in the most suitable cluster to the greatest extent, reducing resource waste and low task execution efficiency caused by improper cluster selection, and improving overall resource utilization efficiency; in the initial cluster allocation, the optimal cluster is given priority, and if resources are insufficient, the suboptimal cluster is tried. After the task is migrated to the suboptimal cluster, the resource status of the optimal cluster is continuously monitored, and secondary migration is triggered when the conditions are met. This dynamic resource allocation strategy can respond to cluster resource changes more flexibly, ensure the resource supply of tasks, and improve the rationality of system resource allocation. The system is robust and efficient. The secondary migration mechanism and cross-cluster resource coordination request settings enable the system to flexibly adjust task deployment based on cluster resource status and task adaptation. When cluster resources are tight or task adaptation changes, the system can respond in a timely manner, enhancing the system's stability and fault tolerance in complex and changing environments and reducing the risk of task failure due to resource issues. The secondary migration priority weight calculation method, combined with the task's adaptability and current priority in different clusters, ensures that high-value, high-priority tasks are preferentially migrated to the optimal cluster for execution when cluster resources permit, thereby ensuring the execution efficiency and effectiveness of key tasks.

[0068] In one possible implementation, the method further includes: If the benefit ratio is detected to be less than the benefit ratio threshold (for example, 0.5) multiple times in a row, an automatic rollback is triggered. For example, if the benefit ratio is detected to be less than the benefit ratio threshold three times in a row, an automatic rollback is triggered. The benefit ratio = (post-migration task execution time - pre-migration task execution time) / migration duration. Global resource rebalancing is performed every k (e.g., 2≤k≤5) scheduling cycles, including: Release inefficient migration caused by resource fragmentation; Recalculate the fitness difference of all in-transit migration tasks; Initiate proactive task recall for clusters whose resource utilization is below the threshold; For tasks that have been stuck in the suboptimal cluster for more than m cycles (e.g., 3 ≤ m ≤ 8), priority decay is applied:

[0069] Among them, b5 is a constant, 0 <b5<1; Score the final priority of the task in the suboptimal cluster. Score the final priority of the task initially in the suboptimal cluster; When the priority after decay is lower than the migration threshold, cross-cluster dependency reorganization is triggered, and the dependency relationship between tasks and clusters and the scheduling strategy are re-evaluated; When the cluster rejects migration requests consecutively for a number of times that reaches the threshold g: Freeze the cluster migration operation for TC = min(2^g × base time, maximum freezing time); The base time may be, for example, 5 minutes, and the maximum freezing time may be, for example, 2 hours; The working principle of the above technical solution is: The effectiveness of task migration is evaluated by calculating the benefit ratio (post-migration task execution time - pre-migration task execution time) / migration duration. If the benefit ratio is detected to be less than 0.5 multiple times (e.g., three times), it indicates that the migration has not achieved the expected results and may even have reduced task execution efficiency. In this case, an automatic rollback is triggered, restoring the task to its pre-migration state to avoid wasting resources on inefficient migration. Global resource rebalancing: Global resource rebalancing is performed every k scheduling cycles to identify and release resources occupied by inefficient migration tasks due to resource fragmentation, and resources are reintegrated for utilization. For all in-progress migration tasks, the fitness difference between them and each cluster is recalculated, and task migration decisions are adjusted based on changes in cluster resource status and task requirements. For clusters whose resource utilization falls below a threshold, a proactive task recall is initiated, migrating some tasks from the low-utilization cluster to improve resource utilization. For tasks that remain in a suboptimal cluster for more than m cycles, a priority decay formula is applied. Over time, the task priority decreases. When the decayed priority falls below the migration threshold, a cross-cluster dependency reorganization is triggered, re-evaluating the task-cluster dependency relationship and scheduling strategy to encourage task migration to a more appropriate cluster. Cluster freeze mechanism: When a cluster rejects migration requests for a consecutive number of times, reaching a threshold g, the cluster freezes the migration duration to prevent resource scheduling disruptions caused by excessive rejections. The base time (e.g., 5 minutes) and maximum freeze time (e.g., 2 hours) are preset values, allowing the cluster time to restore resource status or adjust scheduling strategies.

[0070] The effects of the above technical solution are: The automatic rollback mechanism can promptly correct inefficient migrations, preventing tasks from wasting time and resources in inappropriate clusters, ensuring that tasks are always executed in an efficient environment and improving overall task execution efficiency. Global resource rebalancing enables a more reasonable allocation of resources across clusters by handling inefficient migrations, recalculating fitness differences, and proactively recalling tasks, thereby improving resource utilization and avoiding resource fragmentation and idle waste. Regularly recalculating fitness differences, attenuating the priority of tasks remaining in suboptimal clusters, and restructuring cross-cluster dependencies enable the task scheduling system to dynamically adjust according to cluster resource status and task requirements, better adapting to environmental changes and ensuring the rationality and efficiency of task scheduling. The cluster freezing mechanism prevents the cluster from falling into chaos due to continuous rejection of migration requests. The freezing operation gives the cluster time to adjust, maintains cluster stability, and ensures the reliable operation of the entire system.

[0071] In a possible implementation, the method further includes: When the target cluster resource utilization is lower than the first utilization threshold (determined based on cluster resource planning and business needs) for multiple consecutive periods, the local resource quota of the demotion queue task is increased linearly. The increase ratio is determined by the resource idleness and task priority.

[0072] When the target cluster resource utilization falls below the second utilization threshold (determined by the cluster's minimum resource guarantee requirements), an exponential backoff strategy is used to limit resource allocation frequency to prevent over-allocation and cluster performance degradation. Specifically, the exponential backoff strategy determines the backoff interval based on the difference between resource utilization and the utilization threshold, adjusting the resource allocation interval exponentially.

[0073] The working principle and effects of the above technical solution are as follows: When the target cluster's resource utilization (such as CPU and memory utilization) remains below the first utilization threshold (such as 40%, determined by cluster resource planning and business needs) for multiple consecutive cycles (e.g., three cycles, the cycle duration is configurable), the cluster resources are in a slightly idle state. Tasks currently in the demotion queue (low-priority task queue) are identified and their local resource quotas are increased linearly based on their priority (such as their priority score) and resource idleness. For example, if resource utilization falls below the threshold of 10%, the quota is increased by 5% per cycle until it reaches 30% of the total cluster resources or the quota cap corresponding to the task priority. The linear ratio is calculated as follows: New Quota = Linear Coefficient × (First Threshold - Current Utilization) × Total Cluster Resources, where 0 < Linear Coefficient ≤ 1. The linear coefficient is adjusted dynamically based on task priority (higher-priority tasks have higher linear coefficients). Dynamic adjustment: After each quota increase, cluster utilization is reassessed. If utilization rises above the threshold, the increase is stopped. If utilization remains idle, it is continued to be allocated proportionally until the preset upper limit is reached. The linear increase mechanism ensures that slightly idle resources (above the second threshold and below the first threshold) are rationally utilized by tasks in the downgraded queue, avoiding resource waste.

[0074] When the target cluster resource utilization falls below the second utilization threshold (e.g., 20%, determined by the cluster's minimum resource guarantee requirement), cluster resources are severely idle or unstable, and sudden load shocks caused by frequent resource allocations must be avoided. Execution Logic: The system initiates an exponential backoff strategy. Based on the difference between the current utilization and the second threshold, the system calculates the resource allocation interval (T). This interval increases exponentially with the number of failures (or periods of low utilization). By exponentially extending the allocation interval, inefficient resource requests are reduced, preventing sudden performance drops under sudden loads caused by overallocation.

[0075] The embodiment of the present invention further provides a big data task scheduling system, which is used in the big data task scheduling method described in the embodiment of the present invention. The system includes: The task scoring module is used to obtain task parameters and dependencies between tasks, and calculate task priority scores; The fitness acquisition module is used to determine the fitness of the task and the cluster based on the data locality coefficient of the task, the cluster parameters and the cluster correlation correction coefficient; The task scheduling module is used to determine the target cluster of the task and the order of tasks based on the task priority score and the fitness of the task and the cluster; The circuit breaker migration module is used to trigger the circuit breaker when the number of task failures exceeds the threshold. If the failure is due to cross-cluster dependency, the dependent tasks are migrated to the current cluster and the fitness is recalculated. After continuous success, the priority of the task is restored.

[0076] The principles and effects of the above technical solution are the same as those in the embodiment of the method of the present invention and will not be described in detail here.

[0077] The present invention further provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the methods of the present invention when executing the computer program.

[0078] The present invention is explained from the perspectives of purpose of use, effectiveness, progress and novelty, and meets the functional enhancement and use requirements emphasized by the Patent Law. The above description and drawings of the present invention are only preferred embodiments of the present invention and are not intended to limit the present invention. Therefore, all structures, devices, features, etc. that are similar or identical to the present invention, that is, all equivalent replacements or modifications made in accordance with the scope of the patent application of the present invention, should fall within the scope of protection of the patent application of the present invention.

Claims

1. A task scheduling method for big data, characterized in that: The method comprises: Obtain task parameters and dependencies between tasks, and calculate task priority scores; Determine the fitness of the task and the cluster based on the task's data locality coefficient, cluster parameters, and cluster correlation correction coefficient; Determine the target cluster and sort the tasks based on the task priority score and the fitness of the task with the cluster; When the number of task failures exceeds the threshold, a circuit breaker is triggered. If the failure is due to cross-cluster dependency, the dependent tasks are migrated to the current cluster and the fitness is recalculated. After continuous success, the priority of the task is restored.

2. The task scheduling method for big data according to claim 1, characterized in that: The task parameters include the synchronization time difference of the task, the synchronization success flag, the number of failures of the task, the synchronization cycle, the synchronization time and the current status of the task; Obtain task parameters and dependencies between tasks, and calculate task priority scores, including: Perform interval mapping normalization on each parameter of the task, and obtain the initial priority score of the task by weighted averaging the normalized parameters; Determine a correction factor based on the dependency relationship between tasks; and obtain the first priority score of the task based on the correction factor; ； in, Score the first priority for the task; Score the initial priority of the task; is the correction factor, .

3. The task scheduling method for big data according to claim 1, characterized in that: The cluster parameters include cluster network delay, cluster CPU load, and memory usage; the adaptability of the task and the cluster is determined based on the data locality coefficient of the task, the cluster parameters, and the cluster correlation correction coefficient; including: According to the locality coefficient of the task and the cluster parameters, the basic fitness of the task and the cluster is obtained; ； w1, w2, w3 are the second weight coefficients, w1+ + =1; Determine the priority weight coefficient based on the task priority , combined with the cluster correlation correction coefficient , determine the final fitness of the task and the cluster : ； The cluster correlation correction coefficient is determined based on the inter-cluster network bandwidth, the inter-cluster data interaction frequency, and the historical task scheduling success rate.

4. The task scheduling method for big data according to claim 1, characterized in that: The step of determining the target cluster and sorting the tasks based on the task priority score and the fitness between the task and the cluster includes: If the priority score of a task is lower than the score threshold, and the final fitness of the task with a cluster is greater than b1 times the fitness of the task with the current cluster, then the cluster will be used as the target cluster of the task; b1 1.2; Calculate the final priority score SC of the task in the target cluster based on the task parameters and the target cluster locality gain coefficient new ; ； is the target cluster gain coefficient; ; Score the first priority of the task; determine the target cluster gain coefficient based on the data localization ratio required by the task, the target cluster CPU utilization, and the target cluster memory utilization; If the final task priority score of the task in the target cluster is greater than b2 times the first task priority score of the task in the original cluster, the task is moved to the target cluster; If the final task priority scores of the task in multiple target clusters are all greater than b2 times the first priority score of the task in the original cluster, the cluster with the highest final fitness between the task and the cluster will be selected for migration; and the tasks will be sorted according to their final priority scores in the task list of the final target cluster; b2 1.

2.

5. The task scheduling method for big data according to claim 4, characterized in that: The method further comprises: When multiple tasks are migrated to the same cluster, they are sorted according to the following rules: Score all migration tasks according to their final priority Sort in descending order; if If the data locality coefficients of the tasks are the same, they are sorted in descending order. If the data locality coefficients of the tasks are still the same, they are sorted in ascending order by their arrival time. Before migration, the target cluster resource capacity is checked and migration is performed only if the migration conditions are met. The migration conditions are: ； A collection of tasks to be migrated to the target cluster; is the resource demand of task i in the task set; is the safety margin factor, ranging from [0.1, 0.3].

6. The task scheduling method for big data according to claim 5, characterized in that: The method further comprises: Pre-screen the set of tasks to be migrated and mark them as forced single-cluster tasks; Scoring of forced single-cluster tasks based on final priority Arrange in descending order to generate an independent queue; Tasks in the generated independent queue are submitted to their only target cluster first. If the migration conditions are met, resources are immediately allocated and locked. If not, the cross-cluster resource coordination protocol is triggered to prioritize the resources of tasks in the downgraded queue.

7. The task scheduling method for big data according to claim 6, characterized in that: The method further comprises: If a task has at least one other target cluster that meets the migration conditions in addition to the optimal target cluster, the task is marked as Ta multi ; Initialize the candidate cluster list and sort it in descending order according to the final fitness of each cluster and task; Computing task Ta multi The fitness difference with the optimal cluster and the maximum fitness of the task with other clusters; ； in, is the fitness of the task with the optimal cluster; is the maximum fitness of the task with other clusters; according to Sort in descending order. In the target cluster, tasks with large fitness differences are prioritized. Determine whether the remaining resources of the optimal cluster can meet the current task. If so, assign the migration task to the optimal cluster and lock the task's migration eligibility in other candidate clusters. If not, try the next best cluster. After the task is migrated to the suboptimal cluster, the optimal cluster resource status is continuously monitored. If the following conditions are met at the same time: If the resource utilization of the optimal cluster is less than the utilization threshold for multiple consecutive cycles, the optimal cluster migration condition is met, the ratio of the final fitness of the task with the optimal cluster to the final fitness of the task with the current cluster is greater than b1*b3 times, and the task stays in the suboptimal cluster for ≥b4×synchronization period, then secondary migration is triggered, b3 ≥ 1.1; b4 ≥ 2; The secondary migration priority weight is set as: ； is the secondary migration priority weight, Score the final priority of the task in the current suboptimal cluster; is the fitness of the task and the optimal target cluster; is the fitness of the task with the current suboptimal cluster; When the resource gaps of multiple consecutive candidate clusters exceed the preset ratio of their idle resources, a cross-cluster resource coordination request is triggered.

8. The task scheduling method for big data according to claim 7, characterized in that: The method further comprises: Perform global resource rebalancing every k scheduling cycles; For tasks that have been stuck in the suboptimal cluster for more than m cycles, priority decay is applied: ； Among them, 0 <b5<1; Score the final priority of the task in the suboptimal cluster. Score the final priority of the task initially in the suboptimal cluster; When the priority after decay is lower than the migration threshold, cross-cluster dependency reorganization is triggered, and the dependency relationship between tasks and clusters and the scheduling strategy are re-evaluated.

9. The task scheduling method for big data according to claim 1, characterized in that: The method further comprises: When the target cluster resource utilization rate is lower than the first utilization threshold for multiple consecutive periods, the local resource quota of the degraded queue task is increased in a linear proportion; When the target cluster resource utilization is lower than the second utilization threshold, an exponential backoff strategy is used to limit the resource allocation frequency.

10. A big data task scheduling system, used to implement the big data task scheduling method according to claim 1, characterized in that: The system comprises: The task scoring module is used to obtain task parameters and dependencies between tasks, and calculate task priority scores; The fitness acquisition module is used to determine the fitness of the task and the cluster based on the data locality coefficient of the task, the cluster parameters and the cluster correlation correction coefficient; The task scheduling module is used to determine the target cluster of the task and the order of tasks based on the task priority score and the fitness of the task and the cluster; The circuit breaker migration module is used to trigger the circuit breaker when the number of task failures exceeds the threshold. If the failure is due to cross-cluster dependency, the dependent tasks are migrated to the current cluster and the fitness is recalculated. After continuous success, the priority of the task is restored.

Images