Intelligent dynamic big data platform resource scheduling method and system

Abstract

The application discloses a smart dynamic big data platform resource scheduling method and system, relates to the technical field of intelligent resource management, and comprises the following steps: collecting information of tasks to be scheduled, constructing a directed acyclic graph (DAG), and splitting the tasks according to a load balancing principle; performing topological sorting on the DAG to determine a task priority order, and allocating available resources to the tasks according to the priority order; converting the DAG into a graph neural network input, outputting a priority enhancement value by using a graph convolution network, and re-adjusting the task priority by fusing the priority enhancement value and the sorting according to a preset weight; triggering incremental rescheduling when a new task is added or a task fails, calculating the task priority of a subgraph node affected by the graph convolution network, and performing local reallocation in an original scheduling table. The method disclosed by the application significantly improves the perception and optimization capability of a global task bottleneck, realizes intelligent decision and adaptive adjustment of resource allocation, effectively shortens the total task completion time, and improves the resource utilization rate.

Description

Technical Field

[0001] This invention relates to the field of intelligent resource management technology, specifically to a method and system for resource scheduling on an intelligent dynamic big data platform. Background Technology

[0002] With the rapid development of big data and cloud computing technologies, more and more industries are relying on large-scale distributed computing platforms to efficiently process and analyze multi-source heterogeneous data. Especially in scenarios with extremely high requirements for data timeliness and computing resource utilization, such as finance, telecommunications, transportation, and manufacturing, task scheduling and resource management have become core issues in platform architecture design. In recent years, the emergence of resource scheduling frameworks such as YARN, Kubernetes, and Apache Mesos has enabled the automation and elasticity of cluster management and task scheduling, supporting the efficient operation of multi-tenant and multi-type tasks. Meanwhile, with the increasing application of artificial intelligence and machine learning in big data platform scheduling decisions, scheduling strategies are gradually evolving from traditional static allocation towards intelligent, adaptive, and predictive approaches. For example, some studies have proposed predictive scheduling that combines historical load characteristics, or utilize deep learning to achieve dynamic task grading and optimal resource allocation, further improving the overall system throughput and resource utilization efficiency.

[0003] However, existing big data platform scheduling technologies still have many shortcomings when facing large-scale, dynamic, and heterogeneous computing environments. First, traditional rule-based or static priority-based scheduling methods struggle to adapt to complex dependencies between tasks and fluctuating resource supply. They often fail to promptly identify and optimize critical paths and bottleneck nodes in DAG (Directed Acyclic Graph) task flows, resulting in limited global execution efficiency. Second, current resource scheduling frameworks generally lack comprehensive modeling capabilities for data communication latency, resource heterogeneity, and the dynamic nature of task execution. When faced with the addition or deletion of tasks and adjustments to dependencies during runtime, the response speed of scheduling strategy adjustments is slow and the accuracy is insufficient. Furthermore, while some intelligent scheduling methods have incorporated neural networks or heuristic algorithms, they are mostly offline trained or optimized at single points, failing to achieve real-time adaptation to dynamic task flows and resource pools. Moreover, the scheduling results lack interpretability and are difficult to fine-grainedly adjust for actual bottleneck problems. Summary of the Invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the technical problem solved by this invention is that existing big data platform resource scheduling technologies generally suffer from insufficient modeling of complex task dependencies and resource heterogeneity, limited adaptability to dynamic changes during runtime, and difficulty in meeting the efficient scheduling requirements in large-scale dynamic scenarios.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a resource scheduling method for an intelligent dynamic big data platform, comprising:

[0007] Collect information on tasks to be scheduled, construct a directed acyclic graph (DAG), and split the tasks according to the load balancing principle.

[0008] Perform topological sorting on the DAG to determine the task priority order, and allocate available resources to the tasks according to the priority order;

[0009] The DAG is converted into a graph neural network input, and the graph convolutional network outputs a priority enhancement value. The priority enhancement value is then fused with the ranking according to a preset weight to readjust the task priority.

[0010] Incremental rescheduling is triggered when a new task is added or a task fails. The graph convolutional network locates and calculates the task priority of the affected subgraph nodes and performs local reallocation in the original scheduling table.

[0011] As a preferred embodiment of the intelligent dynamic big data platform resource scheduling method described in this invention, the step of splitting tasks according to the load balancing principle includes: calculating the average execution level of the tasks to be scheduled, setting a splitting threshold based on the average execution level; traversing the tasks in the directed acyclic graph (DAG) to determine whether the average execution time of the task on all resources exceeds the threshold; if it does not exceed the threshold, the task is retained in its original state; if it exceeds the threshold, the task is split into multiple balanced subtasks.

[0012] For each split task, remove the original node in the graph and create multiple new nodes according to the number of subtasks after balanced division; each new node inherits all predecessor dependencies of the original node. If the subtasks must be executed serially, add sequential dependencies between the subtasks in order; otherwise, keep them parallel; set the corresponding execution duration information for each subtask.

[0013] After the split is completed, the DAG is checked for loops, dependency cycles are removed, and the predecessor and successor lists of newly added nodes are updated.

[0014] The information of the tasks to be scheduled includes the execution time, dependencies, and data transmission overhead of each task.

[0015] As a preferred embodiment of the intelligent dynamic big data platform resource scheduling method described in this invention, the topological sorting of the DAG includes performing a traversal search on the DAG to generate a topological sequence that satisfies the dependency constraints.

[0016] Based on the topology sequence, the scheduler calculates the upward ranking of each node from the beginning of the sequence, from bottom to top. For exit nodes without successors, the upward ranking is equal to the average execution time of the task itself. For other nodes, the upward ranking is equal to the average execution time of the task plus the data transmission time with the successor node, plus the upward ranking of the successor node, and the maximum value among all successor node path cases is taken.

[0017] Identify the critical tasks that affect the overall completion time, i.e., the task paths with the highest cumulative upward ranking from the entry point to the exit point; the scheduler arranges all tasks in descending order of upward ranking to obtain the task priority order.

[0018] As a preferred embodiment of the intelligent dynamic big data platform resource scheduling method described in this invention, the method of allocating available resources to tasks includes: making incremental scheduling decisions, determining the earliest possible start time for the current task, adding the estimated runtime required for the current task on the resources to be allocated, and evaluating the completion time; comparing the current task with the available resources, and allocating the current task to the earliest available resource.

[0019] The idle time of the selected resource is updated to the actual completion time of the task, and the start and end times of the current task on the resource are recorded; all tasks are assigned in priority order to obtain the scheduling table.

[0020] As a preferred embodiment of the intelligent dynamic big data platform resource scheduling method described in this invention, the step of using a graph convolutional network to output priority enhancement values ​​includes extracting the average execution time per node on computing resources, the number of predecessor tasks, the number of successor tasks, and the ranking of remaining workload for each node, and combining them into node features after normalization; for each dependent edge, recording the data transmission time and constant flag between the corresponding tasks, and using them as edge features after normalization.

[0021] Construct a multi-layer graph neural network, with each layer including three types of mappings: mapping 1 applies to the node's own features, mapping 2 applies to the features from the predecessor node, and mapping 3 applies to the edge features; all mappings are linear transformations.

[0022] In each layer, a node first performs a linear transformation on its own upper-layer representation;

[0023] Traverse all predecessor nodes, perform a linear transformation on the upper-level representation of each predecessor node, and add a linear transformation corresponding to the edge feature; add the transformation result of the node itself to the transformation results of all predecessors, and obtain the new feature representation of the node's current layer through ReLU; repeat the iteration until the maximum number of iterations, and each node obtains a high-dimensional representation containing its own and its neighbors' computational characteristics and topological information.

[0024] After the last layer of the graph neural network, a set of readout parameters is added to perform a linear mapping on the high-dimensional representation of each node to obtain the priority offset value;

[0025] Set fusion weights to balance the upward ranking and the priority offset value of the GNN output; for each node, the upward ranking of the node is proportionally weighted and summed with the priority offset value to obtain the final scheduling priority score; use the new score to reorder all tasks to obtain a scheduling sequence that can dynamically respond to bottlenecks and key nodes in the graph.

[0026] As a preferred embodiment of the intelligent dynamic big data platform resource scheduling method described in this invention, the triggering of incremental rescheduling includes: deploying the trained GNN model for scheduling services, and monitoring in real time for new tasks, canceled tasks, and modified events of task dependencies; starting from the task nodes and dependencies designed in the events, performing a bounded traversal along the predecessor and successor directions to find tasks that have not yet started execution, and obtaining a set of affected tasks.

[0027] For each affected task, information on execution stability, task connectivity, remaining workload, and edge features is collected; the subgraph corresponding to the affected task is used as a single graph sample and input into the GNN; the priority offset value and resource affinity score of the affected task node are obtained through inference output; the resource affinity score represents the preference score of task resource combination under simulated execution.

[0028] A comprehensive priority is obtained by fusing resource affinity and priority offset values; incremental scheduling decisions are then made for affected tasks based on the comprehensive priority.

[0029] As a preferred embodiment of the intelligent dynamic big data platform resource scheduling method described in this invention, the local reallocation includes: using the difference between the predicted completion time and the actual completion time, and the incremental subgraph as new samples, to perform a small number of gradient updates on the GNN; after each new DAG is formed and updated, the GNN is run forward to update the comprehensive priority of all nodes, and the task start time and resource allocation obtained from the incremental scheduling are input into the execution layer.

[0030] As a preferred embodiment of the intelligent dynamic big data platform resource scheduling system described in this invention, it includes a task management module, a priority calculation module, a HEFT scheduling module, and a dynamic optimization module.

[0031] The task management module is used to collect task information and construct and maintain the DAG structure;

[0032] The priority calculation module is used to perform topology analysis, upward ranking, and priority sorting of the DAG;

[0033] The HEFT scheduling module is used to map tasks to specific resources according to priority and handle communication overhead.

[0034] The dynamic optimization module is used for runtime monitoring, GNN feature extraction, and online model fine-tuning.

[0035] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program as a step in implementing a resource scheduling method for an intelligent dynamic big data platform.

[0036] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of a resource scheduling method for an intelligent dynamic big data platform.

[0037] The beneficial effects of this invention are as follows: The intelligent dynamic big data platform resource scheduling method provided by this invention can achieve efficient and intelligent scheduling for complex task dependencies and variable resource environments. Through DAG task dependency modeling and critical path identification, it significantly improves the ability to perceive and optimize global bottlenecks in tasks; combined with HEFT and GNN-enhanced dynamic priority ranking, it achieves intelligent decision-making and adaptive adjustment of resource allocation, effectively shortening the total task completion time and improving resource utilization. Furthermore, the system supports dynamic addition and deletion of tasks and resources, and incremental scheduling under dependency adjustments, possessing good scalability and real-time response capabilities. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 The first embodiment of the present invention provides an overall flowchart of a resource scheduling method for an intelligent dynamic big data platform. Detailed Implementation

[0040] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0041] Example 1, referring to Figure 1As an embodiment of the present invention, a resource scheduling method for an intelligent dynamic big data platform is provided, comprising:

[0042] S1: Collect information on tasks to be scheduled, construct a directed acyclic graph (DAG), and split the tasks according to the load balancing principle.

[0043] Furthermore, initialize the input by reading the entire task list {T1, T2, ..., T...} from the scheduling system or user interface at once. n} and resource list {R1,R2,…,R K}

[0044] For each task T i Read: On each resource R k Execution time estimation Dependencies with other tasks on {(T) p ,T i Data transfer time between tasks Calculate the average execution time and refer to Makespan.

[0045] For each task T i Calculate its average execution time:

[0046]

[0047] Set global reference Used to determine the threshold for "large tasks". The benchmark can be the sum of the average execution times of all tasks or the longest completion time in historical scheduling.

[0048] Initially construct a DAG, directly using the task set as the node set of the graph:

[0049] V←{T1,T2,…,T n}

[0050] For each dependency (T) p ,T i Add edge set E to edge set E by T p Point to T i The directed edge.

[0051] Task splitting according to the load balancing principle includes: calculating the average execution level of the tasks to be scheduled; setting a splitting threshold based on the average execution level; traversing the tasks in the directed acyclic graph (DAG) to determine whether the average execution time of the task on all resources exceeds the threshold; if it does not exceed the threshold, the task is retained in its original state; if it exceeds the threshold, the task is split into multiple balanced subtasks.

[0052] Large task detection and splitting, traversing each node T i If its average execution time satisfy:

[0053]

[0054] It is considered a "big task" that needs to be broken down.

[0055] For each split task, remove the original node in the graph and create multiple new nodes according to the number of subtasks after balanced division; each new node inherits all predecessor dependencies of the original node. If the subtasks must be executed serially, add sequential dependencies between the subtasks in order; otherwise, keep them parallel; set the corresponding execution duration information for each subtask.

[0056] The information of the tasks to be scheduled includes the execution time, dependencies, and data transmission overhead of each task.

[0057] Calculate the number of smaller tasks after splitting:

[0058]

[0059] Let the execution time of each subtask be approximately

[0060] Remove the original task T from the node set. i Add m new sub-task nodes {T i,1 ,…,T i,m}. The original T i All front-wheel drives point to each T i,j ; each T i,j Points to all previous successors to inherit data dependencies. If it's necessary to ensure subtasks execute in order, then in T... i,1 →T i,2 →…→T i,m Add sequential dependencies between them; otherwise, keep all subtasks in parallel.

[0061] After the split is complete, perform a loop check on the DAG, remove dependency cycles, and update the predecessor and successor lists of newly added nodes.

[0062] For new nodes introduced by the split, if subsequent scheduling deploys them on different resources, the transmission time and graph structure will still use the original task-to-task relationship. Estimate the transfer between subtasks (either amortized or conservatively using the original value). The final output is a new set of nodes V′ and edge set E′ after splitting and dependency inheritance, which is the preprocessed DAG.

[0063] It should be noted that by modeling the dependencies between tasks as a directed acyclic graph (DAG), the execution order and data flow of complex business processes can be accurately described, effectively avoiding deadlocks and conflicts during scheduling. Simultaneously, the automatic task splitting mechanism targets large-scale computing units, rationally dividing them into fine-grained subtasks, thus making fuller use of computing resources and improving the system's parallelism and overall throughput. An intelligent splitting criterion based on the "execution time as a percentage of Makespan" is introduced, enabling dynamic granular adjustment. Task splitting not only optimizes load balancing but also lays the foundation for subsequent adaptive scheduling strategies (such as GNN-based learning reinforcement), significantly improving system elasticity and scheduling accuracy compared to traditional static task configuration methods.

[0064] S2: Perform topological sorting on the DAG to determine the task priority order, and allocate available resources to the tasks according to the priority order.

[0065] Furthermore, the topological sorting of the DAG includes performing a traversal search on the DAG to generate a topological sequence that satisfies the dependency constraints.

[0066] Based on the topology sequence, the scheduler calculates the upward ranking of each node from the beginning of the sequence, working from bottom to top. For exit nodes without successors, the upward ranking is equal to the average execution time of the task itself. For other nodes, the upward ranking is equal to the average execution time of the task plus the data transmission time with successor nodes, plus the upward ranking of the successor nodes, and the maximum value among all successor node path cases is taken.

[0067] Identify the critical tasks that affect the overall completion time, i.e., the task paths with the highest cumulative upward ranking from the entry point to the exit point; the scheduler arranges all tasks in descending order of upward ranking to obtain the task priority order.

[0068] Furthermore, the allocation of available resources for the task includes making incremental scheduling decisions, determining the earliest possible start time for the current task, adding the estimated runtime required for the current task on the resources to be allocated, and evaluating the completion time; comparing the current task with the available resources, and allocating the current task to the earliest available resource.

[0069] The idle time of the selected resource is updated to the actual completion time of the task, and the start and end times of the current task on the resource are recorded; all tasks are assigned in priority order to obtain the scheduling table.

[0070] Furthermore, initialize the data structure for each task node T. i Preparation: Precursor list pred(T) i ): Contains all direct pointers to T i Task; successor list succ(T) i): Contains all from T i The task at hand; accessing the state flag state(T) i The initial value is set to "not accessed".

[0071] Depth-first search (DFS) generates a topology sequence, traversing all task nodes sequentially. If state(T) i If a node is "not visited", then recursive access will begin from that node.

[0072] Recursive access process (for the current node T) i ): will set state(T) i ) is marked as "under access"; for each subsequent task T j ∈succ(T i ): If state(T) j The result is still "not visited", so recursively visit T. j After all successor nodes have completed their visits, T will be... i Add "end of topological sequence"; set state(T) i Mark it as "Completed".

[0073] Finally, the "topology sequence" is the valid scheduling preorder in reverse order of the order in which tasks were added.

[0074] Rank Up The calculation, based on the generated topological sequence, is processed sequentially from the sequence head (exit node) to the end: If task T i There is no successor in the computation graph (i.e., succ(T)). i If the result is empty, then its ranking upwards will be directly set to the average execution time of that task.

[0075]

[0076] Otherwise, for all successors T j ∈succ(T i Take the maximum value in the following formula:

[0077]

[0078] in, It is task T i Average execution time across all resources It is T i →T j Data transmission time.

[0079] Determine the critical path and task priorities, and perform critical path identification; the critical path of the entire DAG is the maximum sum of the critical paths from a certain entry task to a certain exit task. The value of that path is the lower bound of the minimum possible completion time globally.

[0080] Task sorting, by each task The values ​​are sorted from largest to smallest to obtain a priority list of tasks. In the subsequent resource allocation phase, the scheduler will prioritize tasks with higher priority (those with a greater impact on Makespan).

[0081] Enumerating the resource set lists all available computing resources in the system as a set:

[0082] R = {R1, R2, ..., R} K}

[0083] Set the initial availability time for each resource R k Record its current "available time":

[0084] avail(R k ) = 0

[0085] This value represents the earliest idle time when no task has been assigned to this resource.

[0086] Obtain the task scheduling list, and extract an ordered list from the obtained task priority sorting:

[0087]

[0088] in Has the highest priority (maximum) ), decreasing sequentially.

[0089] Assign tasks according to priority, for each task T in list L. i Perform the following operations:

[0090] Prepare for the completion time of the precursor, if T i There is a set of predecessors, pred(T) i For each predecessor T, p Its completion time finish(T) has been recorded. p ).

[0091] Calculate EST and EFT for each resource R. k ∈R is used for calculation. The earliest start time formula is expressed as:

[0092]

[0093] Where avail(R) k ) represents the idle time of the resource itself; if the predecessor T p Assigned to T i If the resources are the same, the corresponding communication time can be omitted.

[0094] The earliest completion time formula is expressed as:

[0095]

[0096] in It is T i In R k Execution time estimation.

[0097] The optimal resource is selected by finding the resource index that minimizes the completion time. The formula is as follows:

[0098]

[0099] T i Assigned to Record the scheduling results and update the resource status; record the task start time, expressed by the formula:

[0100]

[0101] The completion time formula is expressed as follows:

[0102]

[0103] Updated resource availability time:

[0104]

[0105] The output scheduler is repeated in the above iterations until all T are reached. i All ∈L are assigned.

[0106] The final output is for each task T. i Resource allocation start time start(T) i ) and completion time finish(T) i ).

[0107] It should be noted that combining the critical path method with topology sorting allows for precise identification of system bottlenecks and scheduling priorities from a global perspective, ensuring the priority execution of critical tasks, effectively shortening the overall completion time (Makespan), and improving overall scheduling efficiency. Topology sorting also ensures strict satisfaction of dependency constraints, avoiding unnecessary waiting and conflicts. Based on traditional topology sorting, the "upward ranking" algorithm of the critical path method is integrated to quantify the global contribution of tasks to the final completion time. This hierarchical recursive ranking mechanism provides, for the first time, a theoretically rigorous dynamic basis for priority allocation in DAG scheduling scenarios, and provides a structured input feature foundation for introducing machine learning enhancements into subsequent algorithms.

[0108] S3: Convert the DAG into a graph neural network input, use the graph convolutional network to output priority enhancement values, and merge the priority enhancement values ​​with the ranking according to preset weights to readjust the task priority.

[0109] Furthermore, the priority enhancement values ​​output by the graph convolutional network include extracting the average execution time per node, the number of predecessor tasks, the number of successor tasks, and the ranking of remaining workload on computing resources for each node, and combining them into node features after normalization; for each dependent edge, the data transmission time and constant label between the corresponding tasks are recorded, and normalized as edge features.

[0110] Construct a multi-layer graph neural network, with each layer including three types of mappings: mapping 1 applies to the node's own features, mapping 2 applies to the features from the predecessor node, and mapping 3 applies to the edge features; all mappings are linear transformations.

[0111] In each layer, a node first performs a linear transformation on its own upper-layer representation. It then iterates through all predecessor nodes, performing a linear transformation on the upper-layer representation of each predecessor node, and adding a linear transformation corresponding to the edge feature. The node's own transformation result is added to the transformation results of all predecessor nodes, and ReLU is used to obtain the new feature representation of the node in this layer. This process is repeated until the maximum number of iterations is reached, resulting in a high-dimensional representation for each node containing its own computational characteristics and topological information, as well as information about its neighbors.

[0112] After the last layer of the graph neural network, a set of readout parameters is added to perform a linear mapping on the high-dimensional representation of each node, resulting in a priority offset value.

[0113] Set fusion weights to balance the upward ranking and the priority offset value of the GNN output; for each node, the upward ranking of the node is proportionally weighted and summed with the priority offset value to obtain the final scheduling priority score; use the new score to reorder all tasks to obtain a scheduling sequence that can dynamically respond to bottlenecks and key nodes in the graph.

[0114] Furthermore, the node feature vector for each task node T i Construct the initial feature vector:

[0115]

[0116] in, This represents the average execution time of the task across all resources. |pred(T i )|,|succ(T i | represents the number of predecessor / successor tasks (discrete numerical value, which can be normalized). This indicates ranking upwards.

[0117] The edge feature vector for each directed dependent edge (T) p →Ti ), Construction:

[0118]

[0119] Indicates the data transmission time (normalized), and "1" indicates the weight placeholder for a single dependency.

[0120] Design the GNN encoder network topology. The number of layers is L, and the hidden dimension of each layer is d. Use graph convolutional or graph attention (GAT) layers with edge features.

[0121] Message passing rules, let the k-th level node be represented as Layer 0 is... For each level k = 1…L, execute:

[0122]

[0123] in, Let be the weight matrix to be learned. σ(·) is the ReLU activation. After the Lth layer, the output readout layer performs a linear mapping on each node to obtain the "priority score offset":

[0124]

[0125] in, Node-priority score prediction. Meaning of offset δ. i This indicates that GNN is used for task T i The priority increment "relative to traditional ranking".

[0126] δ i >0 indicates that GNN believes the task should be given higher priority. i <0 indicates that its priority can be appropriately reduced. Normalization process will apply to all {δ} i Mapping to the [-1,1] interval ensures numerical stability.

[0127] Priority fusion and scheduling reordering: The fusion formula introduces a fusion coefficient α∈[0,1], combining the classic upward ranking with the GNN output.

[0128]

[0129] in, Represents the standardized δ i 2 Rescheduling is based on priority(T) i A new task queue is generated in descending order and passed to the HEFT allocation process to dynamically respond to bottlenecks and topology criticality.

[0130] Model training and online inference: The training set is constructed by sampling multiple labeled DAG instances from historical DAG scheduling logs.

[0131] Each node provides the actual scheduling "order" or its contribution to the global Makespan (as the regression label y). i ).

[0132] The loss function formula is expressed as:

[0133]

[0134] The first term is the mean squared error, and the second term is the L2 regularization (weight decay coefficient λ).

[0135] Once the validation set error no longer decreases, the trained GNN encoder is loaded into the scheduling service. After each DAG construction or update, a forward propagation is performed, and all δ values ​​are computed. i , merge to generate a new scheduling priority queue

[0136] It should be noted that by evaluating the earliest start and earliest finish times of tasks in parallel on each resource and combining this with real-time information from the dynamically available resource pool, adaptive optimal matching of tasks and resources is achieved. Unlike simple greedy or static allocation strategies, this method effectively accommodates heterogeneous environments and task dependency complexity, improving the intelligence and adaptability of scheduling.

[0137] S4: Incremental rescheduling is triggered when a new task is added or a task fails. The graph convolutional network locates and calculates the task priority of the affected subgraph nodes and performs local reallocation in the original scheduling table.

[0138] Furthermore, the triggering of incremental rescheduling includes deploying the trained GNN model for scheduling services, and monitoring in real time for events such as adding tasks, canceling tasks, and modifying task dependencies; starting from the task nodes and dependencies designed in the events, performing a bounded traversal along the predecessor and successor directions to find tasks that have not yet started execution, thereby obtaining a set of affected tasks.

[0139] For each affected task, information on execution stability, task connectivity, remaining workload, and edge features is collected. The subgraph corresponding to the affected task is used as a single graph sample and input into the GNN. The priority offset value and resource affinity score of the affected task node are obtained through inference output. The resource affinity score represents the preference score of task resource combination under simulated execution.

[0140] A comprehensive priority is obtained by fusing resource affinity and priority offset values; incremental scheduling decisions are then made for affected tasks based on the comprehensive priority.

[0141] The local reallocation includes using the difference between the predicted completion time and the actual completion time, and the incremental subgraph as new samples, to perform a small number of gradient updates on the GNN; after each new DAG is formed and updated, the GNN is run forward to update the overall priority of all nodes, and the task start time and resource allocation obtained from the incremental scheduling are input into the execution layer.

[0142] Furthermore, in the initial subgraph localization, three cases are handled: When a new task is added, its metadata {ET, pred, succ} is captured; when a task fails or is canceled, the failed task T is captured. f and its allocated resources R fail .

[0143] Change the relevant node set ΔV={T new} or {T f Using} as the seed, perform a breadth-first traversal of finite depth d along the successor direction of the DAG to collect all affected tasks that have not started or are in the queue, forming a subgraph node set A.

[0144] At the same time, the edge set E of these nodes in the original DAG is retained. A .

[0145] Node feature vector x i Original characteristics include average execution time. Front-wheel drive / Successor | pred(T) i )|,|succ(T i )|, original upward ranking

[0146] Event identifiers: newly added tasks are marked with a vector "+1", failed tasks are marked with "1", and the rest are marked with 0.

[0147] Edge features, data transmission time Alternatively, simply assigning a value of 1 indicates that a dependency exists.

[0148] GCN forward reasoning and priority re-evaluation. Constructing the adjacency matrix A. A For subgraph G[A], construct the normalized adjacency matrix with self-loops:

[0149]

[0150] Perform multi-layer graph convolution to initialize the hidden representation. The formula for the l-th layer is expressed as:

[0151]

[0152] Where σ is ReLU, W (l) These are trainable weights. The priority score output is taken from the last layer's representation H.(L) The new priority enhancement value for each node is calculated through a fully connected readout layer:

[0153]

[0154] δ i Normalized to [0,1] as the GCN score.

[0155] Integrate traditional ranking for each affected node T i For ∈A, calculate the fusion priority, expressed by the formula:

[0156]

[0157] Here, α∈[0,1] controls the weight of the CPM heuristic and the GCN score.

[0158] Local sorting, by priority ′ Reorder the tasks in A from largest to smallest to generate a new local scheduling sequence S. A Local resource reallocation and scheduling table updates are performed, while the locked area retains the resource allocation and start / complete times of tasks that have already started or completed in the original scheduling table.

[0159] Perform HEFT local scheduling, press S A In the order of tasks T i Enumerating resources R k The earliest start / finish time of the increment is calculated using the following formula:

[0160]

[0161] Select R that minimizes EFT k And allocate, update avail(R) k The scheduling table is partially merged, and S is... A The new allocation results are written back to the global scheduling table, overwriting the old A region entries. Unaffected task entries remain unchanged.

[0162] The scheduling system supports real-time changes to tasks and dependencies at runtime, greatly enhancing the platform's flexibility and robustness. It can adaptively handle task additions, cancellations, or dependency adjustments, maintaining consistency and optimality in resource allocation and task execution, making it suitable for large-scale distributed and dynamic business scenarios.

[0163] A DAG dynamic scheduling scheme is based on "incremental adjustment" and "online GNN fine-tuning." Unlike global recalculation, the system only locally rearranges the affected subgraphs, significantly reducing scheduling overhead. The GNN model supports small-step online learning, quickly adapting to topology changes. This combination not only ensures real-time and optimal scheduling but also promotes the development of intelligent scheduling systems towards self-adaptation and self-evolution.

[0164] It should be noted that by automatically learning the latent scheduling patterns in the DAG structure through graph neural networks (GNNs), bottleneck nodes and critical path tasks can be dynamically identified, enabling intelligent enhancement of task priority. With the accumulation of historical data, the model continuously evolves, adapting to different types of task structures and resource states, making the scheduling strategy more refined and personalized. By modeling DAG scheduling as the input to the graph neural network and combining multi-dimensional features of tasks and dependencies, end-to-end training enhances the priority output. This hybrid scheduling scheme, integrating deep learning and classical heuristics, breaks through the limitations of previous scheduling methods relying solely on static rules or single indicators, achieving a deep integration of structural cognition and decision optimization.

[0165] Example 2 is an embodiment of the present invention, which provides a resource scheduling method for an intelligent dynamic big data platform. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0166] First, this embodiment takes the scientific computing task scheduling scenario of a big data computing center as the background. The system contains 20 heterogeneous computing nodes and needs to complete a batch of scientific tasks with multiple dependencies. In actual operation, the platform needs to achieve high throughput and high concurrency automatic scheduling of multiple tasks, adapting to dynamic changes in task distribution, load, and resource status. To objectively compare the advantages and disadvantages of the intelligent dynamic scheduling method and the traditional heuristic scheduling, the existing earliest completion time first (EFT) method and the "DAG-load balancing-topology GNN enhancement-incremental scheduling" combination method described in this invention are used to perform complete scheduling and execution on the same batch of tasks.

[0167] The detailed experimental procedure is as follows:

[0168] The platform receives 40 scientific computing tasks in batches, providing the execution time range for each task ([3,45] minutes), and collects the dependencies and data transmission overhead between each task (ranging from 0 to 5 minutes). A Directed Acyclic Graph (DAG) is automatically constructed using the task information to form a task network.

[0169] The split DAG is topologically sorted, and the upward ranking of each task node is calculated according to the bottom-up principle (average execution time plus maximum data transmission delay), forming a global priority sequence to determine bottleneck nodes and critical paths.

[0170] The platform encodes the global DAG using a three-layer graph convolutional network (GCN) based on node features (average duration, dependency, remaining workload, etc.) and edge features (data transmission latency), outputting node priority enhancement values. The GNN output and upward ranking are then fused with a weighted average of 0.6:0.4 to reorder task priorities and dynamically strengthen bottleneck task scheduling.

[0171] During execution, the system randomly injects disturbances such as task additions and task failures, triggering an incremental scheduling process. GNN quickly locates the affected subgraphs and calculates their priorities and resource affinity. The platform only partially reallocates resources for the affected tasks, while unaffected tasks retain their existing allocations.

[0172] Record key metrics such as overall completion time, CPU / GPU utilization at each stage, task waiting time, and rescheduling delay caused by sudden changes, and compare them with traditional EFT methods.

[0173] Regarding total completion time, traditional EFT algorithms, relying solely on the earliest task completion strategy, struggle to handle concentrated loads on bottleneck nodes or critical path tasks, easily leading to both idle resources and critical task blocking. By employing the DAG-GNN method of this invention, tasks are rationally split, critical path identification is more accurate, and GNN further enhances the perception and optimization of dependencies and bottleneck nodes, reducing the global task completion time from 138 minutes to 122 minutes (a 17.2% improvement). When new tasks are added or execution fails, this invention's method maintains global Makespan improvement through subgraph localization and local reordering, while traditional solutions suffer from global rescheduling, resulting in a significant increase in completion time.

[0174] Regarding resource utilization, traditional EFT allocation strategies fail to fully consider task splitting and dynamic resource affinity. This invention, however, proactively splits large tasks through load balancing principles and leverages GNN feature learning to effectively improve global and local parallelism, resulting in a stable resource utilization rate exceeding 78%.

[0175] The average task waiting time is significantly reduced. Especially under dynamic disturbance events, the intelligent scheduling method shortens the waiting period of affected tasks from 28 minutes to about 14 minutes through priority rearrangement and local resource reallocation, improving efficiency by nearly 50%.

[0176] The increase in the number of critical path tasks reflects that, after GNN optimization, the scheduling system can proactively identify and enhance the parallelism of bottleneck paths. The GNN output priority adjustment number reaches 13-14 times, indicating that the system can dynamically identify bottlenecks and accelerate tasks based on graph topology and node characteristics, effectively making up for the shortcomings of classic algorithms in "limited ability to perceive global dependencies and parallelism".

[0177] Example 3, an embodiment of the present invention, provides an intelligent dynamic big data platform resource scheduling system, including a task management module, a priority calculation module, a HEFT scheduling module, and a dynamic optimization module.

[0178] The task management module is used to collect task information and construct and maintain the DAG structure.

[0179] The priority calculation module is used to perform topology analysis, upward ranking, and priority sorting on the DAG.

[0180] The HEFT scheduling module is used to map tasks to specific resources according to priority and handle communication overhead.

[0181] The dynamic optimization module is used for runtime monitoring, GNN feature extraction, and online model fine-tuning.

[0182] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0183] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0184] More specific examples (a non-exhaustive list) of computer-readable media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0185] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc. It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

[0186] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A resource scheduling method for an intelligent dynamic big data platform, characterized in that, include: Collect information on tasks to be scheduled, construct a directed acyclic graph (DAG), and split the tasks according to the load balancing principle. For each split task, remove the original node in the graph and create multiple new nodes according to the number of subtasks after balanced division; Perform topological sorting on the DAG to determine the task priority order, and allocate available resources to the tasks according to the priority order; The DAG is converted into a graph neural network (GNN) input, and a priority offset value is output using a graph convolutional network. The priority offset value is then fused with the sorting algorithm according to preset weights to readjust the task priorities, including: The graph neural network outputs the priority offset value of each node. A fusion weight is set, and for each node, the node's upward ranking and priority offset value are weighted and summed according to the set fusion weight to obtain the final scheduling priority score. The scheduling priority score is used to reorder all tasks to determine the readjusted task priority order. The upward ranking is obtained by performing a traversal search on the DAG to generate a topological sequence that satisfies the dependency constraints; Based on the topology sequence, the scheduler calculates the upward ranking of each node from the beginning of the sequence, from bottom to top. For exit nodes without successors, the upward ranking is equal to the average execution time of the task itself. For other nodes, the upward ranking is equal to the average execution time of the task plus the data transmission time with the successor node, plus the upward ranking of the successor node, and the maximum value among all successor node path cases is taken. Incremental rescheduling is triggered when a new task is added or a task fails. The graph convolutional network locates and calculates the task priority of the affected subgraph nodes and performs local reallocation in the original scheduling table.

2. The intelligent dynamic big data platform resource scheduling method as described in claim 1, characterized in that: The task splitting according to the load balancing principle includes calculating the average execution level of the tasks to be scheduled and setting a splitting threshold based on the average execution level. Traverse the tasks in the directed acyclic graph (DAG) and determine whether the average execution time of the task on all resources exceeds the threshold. If the time limit is not exceeded, the task will remain as is. If the number of tasks exceeds the limit, the task will be divided into multiple balanced subtasks. For each split task, remove the original node in the graph and create multiple new nodes according to the number of subtasks after balanced division; each new node inherits all predecessor dependencies of the original node. If the subtasks must be executed serially, add sequential dependencies between the subtasks in order; otherwise, keep them parallel; set the corresponding execution duration information for each subtask. After the split is completed, the DAG is checked for loops, dependency cycles are removed, and the predecessor and successor lists of newly added nodes are updated. The information of the tasks to be scheduled includes the execution time, dependencies, and data transmission overhead of each task.

3. The intelligent dynamic big data platform resource scheduling method as described in claim 2, characterized in that: The topological sorting of the DAG includes performing a traversal search on the DAG to generate a topological sequence that satisfies the dependency constraints. Based on the topology sequence, the scheduler calculates the upward ranking of each node from the bottom up, starting from the head of the sequence; for exit nodes without successors, the upward ranking is equal to the average execution time of the task itself. For other nodes, the upward ranking is equal to the average execution time of the task plus the data transfer time with the successor node, plus the upward ranking of the successor node, and the maximum value among all successor node path cases is taken. Identify the critical tasks that affect the overall completion time, i.e., the task paths with the highest cumulative upward ranking from the entry point to the exit point; the scheduler arranges all tasks in descending order of upward ranking to obtain the task priority order.

4. The intelligent dynamic big data platform resource scheduling method as described in claim 3, characterized in that: The allocation of available resources to the task includes making incremental scheduling decisions, determining the earliest possible start time for the current task, adding the estimated runtime required for the current task on the resources to be allocated, and evaluating the completion time. Compare the current task with the available resources and assign the current task to the resource that can be completed earliest. The idle time of the selected resource is updated to the actual completion time of the task, and the start and end times of the current task on the resource are recorded; all tasks are assigned in priority order to obtain the scheduling table.

5. The intelligent dynamic big data platform resource scheduling method as described in claim 4, characterized in that: The priority offset value output by the graph convolutional network includes extracting the average execution time per node on computing resources, the number of predecessor tasks, the number of successor tasks, and the ranking of remaining workload for each node, and combining them into node features after normalization. For each dependent edge, record the data transmission time and constant flag between the corresponding tasks, and use the normalized data as edge features; Construct a multi-layer graph neural network, with each layer including three types of mappings: mapping one applies to the node's own features, mapping two applies to the features from the predecessor node, and mapping three applies to the edge features. All mappings are linear transformations; In each layer, a node first performs a linear transformation on its own upper-layer representation; Traverse all predecessor nodes, perform a linear transformation on the upper-level representation of each predecessor node, and add a linear transformation corresponding to the edge feature; add the transformation result of the node itself to the transformation results of all predecessors, and obtain the new feature representation of the node's current layer through ReLU; repeat the iteration until the maximum number of iterations, and each node obtains a high-dimensional representation containing its own and its neighbors' computational characteristics and topological information. After the last layer of the graph neural network, a set of readout parameters is added to perform a linear mapping on the high-dimensional representation of each node, resulting in a priority offset value.

6. The intelligent dynamic big data platform resource scheduling method as described in claim 5, characterized in that: The triggering of incremental rescheduling includes deploying the trained GNN model for scheduling services and real-time monitoring of events such as new task additions, canceled tasks, and task dependency modifications. Starting with the task nodes and dependencies designed in the event, a bounded traversal is performed along the predecessor and successor directions to find tasks that have not yet started execution, thus obtaining the set of affected tasks; among them, task execution failure includes task cancellation and task dependency modification. For each affected task, information on execution stability, task connectivity, remaining workload, and edge features is collected; the subgraph corresponding to the affected task is used as a single graph sample and input into the GNN; the priority offset value and resource affinity score of the affected task node are obtained through inference output; the resource affinity score represents the preference score of task resource combination under simulated execution. A comprehensive priority is obtained by fusing resource affinity scores and priority offset values; incremental scheduling decisions are then made for affected tasks based on the comprehensive priority.

7. The intelligent dynamic big data platform resource scheduling method as described in claim 6, characterized in that: The local reallocation includes using the difference between the predicted completion time and the actual completion time, and the incremental subgraph as new samples, to perform a small gradient update on the GNN. After each new DAG is formed and updated, the GNN is run forward to update the overall priority of all nodes, and the task start time and resource allocation obtained from incremental scheduling are input into the execution layer.

8. A system employing the intelligent dynamic big data platform resource scheduling method as described in any one of claims 1 to 7, characterized in that: It includes a task management module, a priority calculation module, a HEFT scheduling module, and a dynamic optimization module; The task management module is used to collect task information and construct and maintain DAG structures; The priority calculation module is used to perform topology analysis on the DAG, determine the upward ranking and priority order; The HEFT scheduling module is used to map tasks to specific resources according to priority and to handle communication overhead. The dynamic optimization module is used for GNN feature extraction and online fine-tuning of the GNN model.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the intelligent dynamic big data platform resource scheduling method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the intelligent dynamic big data platform resource scheduling method as described in any one of claims 1 to 7.

Images