Intelligent resource scheduling method and system based on dynamic data bloodline graph

Abstract

The application discloses an intelligent resource scheduling method and system based on a dynamic data blood relationship graph, and relates to the technical field of resource scheduling.The method comprises the following steps: collecting the execution log and flow metadata of tasks in a computing platform in real time, and constructing a dynamic directed weighted blood relationship graph; calculating the blood relationship influence coefficient of each node in the dynamic directed weighted blood relationship graph; obtaining a to-be-scheduled task, and calculating a comprehensive priority score based on the dependency weight of the dynamic blood relationship graph of the to-be-scheduled task, the real-time load state of a target node and the scheduling execution time delay; performing priority sorting on the to-be-scheduled task based on the comprehensive priority score, and allocating cluster resources; and predicting the load trend of the target node, and triggering a migration decision when it is detected that the predicted load of the target node exceeds a threshold value and the key blood relationship task weight proportion borne by the target node exceeds a preset threshold value. Through the dynamic blood relationship graph and the intelligent scheduling algorithm, the efficiency and fairness of resource allocation are realized, the cluster utilization rate is improved, and the task delay is reduced.

Description

Technical Field

[0001] This invention relates to the field of resource scheduling technology, and in particular to an intelligent resource scheduling method and system based on dynamic data lineage graphs. Background Technology

[0002] Currently, in distributed computing environments, traditional resource scheduling methods generally suffer from the following technical shortcomings: First, existing scheduling strategies typically make decisions based solely on static task attributes (such as preset priorities or resource requirements), failing to fully consider the complex data dependencies between tasks. This results in resource allocation failing to reflect the actual impact of tasks on the global lineage graph. Second, the prediction accuracy of dynamic load changes and task execution latency is insufficient, making it difficult to proactively trigger resource rebalancing before overload, leading to low cluster resource utilization or a surge in task latency. Third, migration decision mechanisms lack quantitative evaluation of task lineage weights, which may incorrectly migrate critical lineage tasks under overload scenarios, resulting in decreased system stability. For example, some solutions use the PageRank algorithm to calculate task importance, but its static graph modeling cannot adapt to real-time changes in task dependencies, and the lack of a backpropagation mechanism leads to a one-sided assessment of lineage impact. Other solutions use fixed-weight multi-objective optimization functions for priority scoring, but the weight parameters cannot be dynamically adjusted with the cluster state, resulting in unbalanced resource allocation under high load scenarios. Furthermore, traditional migration strategies rely on a single trigger condition—load threshold—without considering task lineage weighting analysis. This can easily lead to missed opportunities for ensuring critical task resources by blindly migrating tasks with low lineage. These issues collectively make it difficult for existing technologies to achieve efficient and fair resource scheduling optimization in dynamic load environments. Summary of the Invention

[0003] In view of this, this invention proposes an intelligent resource scheduling method and system based on dynamic data lineage graphs, which can improve cluster resource utilization and task scheduling reliability. This invention provides the following technical solution:

[0004] A smart resource scheduling method based on dynamic data lineage graphs, the method comprising:

[0005] The system collects execution logs and stream metadata of tasks in the computing platform in real time, and constructs a dynamic directed weighted lineage graph based on the execution logs and stream metadata.

[0006] The bloodline influence coefficient of each node in the dynamic directed weighted bloodline graph is calculated using a weighted propagation model of forward and reverse dependency paths.

[0007] Obtain the tasks to be scheduled and calculate a comprehensive priority score based on the dependency weights of their dynamic lineage graph, the real-time load status of the target node, and the scheduling execution latency;

[0008] Based on the comprehensive priority score, the tasks to be scheduled are prioritized and then cluster resources are allocated according to the prioritization results.

[0009] Predict the load trend of the target node. When the predicted load of the target node exceeds the threshold and the weight ratio of the key lineage tasks it carries exceeds the preset threshold, a migration decision is triggered.

[0010] Optionally, the real-time acquisition of task execution logs and stream metadata in the computing platform, and the construction of a dynamic directed weighted lineage graph based on the execution logs and stream metadata, includes:

[0011] The execution logs and streaming metadata of the task are captured in real time through a preset time window mechanism. Based on the execution logs and streaming metadata, a multi-dimensional feature vector is extracted. The multi-dimensional feature vector includes at least the data volume, data transmission latency, and task dependency strength.

[0012] Based on the aforementioned multidimensional feature vectors, the dynamic weights of the kinship graph edges are calculated using a weighted combination model. The calculation formula is as follows: ,in, For the source node To the target node The amount of data in the data dependency relationship, For the source node To the target node Transmission delay, For the source node To the target node Dependence strength, , , and The adaptive attenuation coefficient;

[0013] Using nodes as vertices of the kinship graph and data dependencies as directed edges, a weighted kinship graph is formed through the aforementioned dynamic weights.

[0014] Optionally, the calculation of the kinship influence coefficient of each node in the dynamic directed weighted kinship graph using a weighted propagation model of forward and reverse dependency paths includes:

[0015] For nodes All downstream nodes Based on edge weights in the kinship graph With preset task level coefficient The sum of the products is used to calculate the forward propagation effect value;

[0016] For nodes All upstream nodes Based on edge weights in the kinship graph With preset task level coefficient The sum of the power functions of the product is used to calculate the backpropagation effect value;

[0017] Through the preset backpropagation gain coefficient and exponential decay factor The forward propagation influence value and the backward propagation influence value are weighted and combined to obtain the node lineage influence coefficient. The calculation formula is: ,in For nodes The set of downstream nodes, For nodes The set of upstream nodes, and This is the preset task level coefficient.

[0018] Optionally, the step of obtaining the task to be scheduled and calculating a comprehensive priority score based on its dynamic lineage graph dependency weights, the real-time load status of the target node, and the scheduling execution latency includes:

[0019] Obtain the lineage influence coefficient of the task to be scheduled. The current load ratio of the target node Scheduling execution delay and task flexibility coefficient parameter;

[0020] The bloodline weight is dynamically updated using a preset weight adjustment model. Load weight and elastic weight ;

[0021] The parameters and weights are fused using a multi-objective optimization function to calculate the comprehensive priority score, as shown in the formula: ,in, This is the time delay decay factor.

[0022] Optionally, the step of prioritizing the tasks to be scheduled based on the comprehensive priority score and allocating cluster resources according to the ranking results includes:

[0023] Sort the tasks to be scheduled in descending order of their overall priority scores to generate a priority queue. ,in This is the highest priority task.

[0024] Based on the total cluster resources And task priority queues, dynamically set resource allocation thresholds, including: when the overall priority score is greater than or equal to a preset score judgment threshold. At that time, set a high-priority resource threshold. When the overall priority score is greater than or equal to the preset score judgment threshold and less than At that time, set the threshold for medium-optimal resources. When the overall priority score is less than Low-priority resource threshold ;

[0025] Resources are allocated to tasks sequentially according to the priority queue: when Set the task as a high-priority task and allocate resources to it. ,in, For nodes The overall priority score, The total score for all tasks in the queue, when Set the task as a medium-optimal task and allocate resources to it. ,when Allocate resources for low-priority tasks ,in, The total amount of resources allocated to high-priority and medium-priority tasks.

[0026] Optionally, the predicted load trend of the target node, when it is detected that the predicted load of the target node exceeds a threshold and the weight ratio of the key lineage tasks it carries exceeds a preset threshold, triggers a migration decision including:

[0027] A hybrid model based on ensemble deep learning and statistical models predicts the future load of target nodes. The prediction function is: ,in, The privacy-weighted payload prediction value output by the deep learning model. For dynamic mixing coefficients, For constant terms, and The order of the statistical model. and The autoregressive and moving average coefficients of the mixture model are given. Historical load values For historical residuals;

[0028] The predicted load and key lineage task weight ratio of the target node are detected. The formula for calculating the key lineage task weight ratio is as follows: ,in A collection of key bloodline missions. For the target node The current set of tasks being carried out For the target node The total weight of key bloodline missions For the target node The total weight of all tasks;

[0029] If the predicted load of the target node exceeds a preset threshold and the weight ratio of the key lineage task exceeds a preset threshold, then the migration task and the target migration node are selected based on the migration utility function and the benefit amplification mechanism.

[0030] This invention further discloses an intelligent resource scheduling system based on dynamic data lineage maps, comprising:

[0031] The data acquisition module is used to collect the execution logs and stream metadata of tasks in the computing platform in real time, and to construct a dynamic directed weighted lineage graph based on the execution logs and stream metadata.

[0032] The bloodline influence coefficient calculation module is used to calculate the bloodline influence coefficient of each node in the dynamic directed weighted bloodline graph through a weighted propagation model of forward dependency path and reverse dependency path;

[0033] The comprehensive priority score calculation module is used to obtain the task to be scheduled and calculate the comprehensive priority score based on the dependency weight of its dynamic lineage graph, the real-time load status of the target node, and the scheduling execution delay.

[0034] The resource allocation module is used to prioritize the tasks to be scheduled based on the comprehensive priority score, and allocate cluster resources according to the ranking result.

[0035] The load migration module is used to predict the load trend of the target node. When it is detected that the predicted load of the target node exceeds the threshold and the weight ratio of the key lineage tasks it carries exceeds the preset threshold, a migration decision is triggered.

[0036] The present invention further discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0037] The present invention further discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described method.

[0038] The present invention further discloses a computer program product, including a computer program that implements the above-described method when executed by a processor.

[0039] According to the technical solution of this invention, a dynamic directed weighted lineage graph is constructed by real-time collection of task execution logs and streaming metadata. Combined with a weighted propagation model of forward and reverse dependency paths, the node lineage influence coefficient is calculated, overcoming the limitations of traditional static lineage analysis and unidirectional propagation. This allows task importance assessment to dynamically reflect real-time changes in data dependencies. The method constructs a comprehensive priority scoring system with dynamic weight allocation through multi-dimensional parameter fusion, achieving adaptive matching between resource scheduling priority and cluster status. While solving the resource skew problem caused by traditional fixed weight allocation, it significantly improves scheduling flexibility under high load scenarios. Furthermore, through a dual-condition triggering mechanism of load trend prediction and the weight ratio of critical lineage tasks, combined with an optimization algorithm of migration utility function and benefit amplification coefficient, when the predicted load of the target node exceeds a threshold and the weight ratio of its critical lineage tasks exceeds the limit, low-lineage-impact tasks are migrated first, and migration paths are dynamically selected. This effectively avoids the damage to critical task resource guarantees caused by blind migration, thereby improving cluster resource utilization while reducing task latency, achieving synergistic optimization of resource scheduling efficiency and fairness. Attached Figure Description

[0040] For illustrative and not limiting purposes, the present invention will now be described in conjunction with embodiments and accompanying drawings, wherein:

[0041] Figure 1 This is a flowchart illustrating the intelligent resource scheduling method based on dynamic data lineage graphs in an embodiment of the present invention.

[0042] Figure 2 This is a flowchart illustrating the intelligent resource scheduling system based on dynamic data lineage graphs in an embodiment of the present invention.

[0043] Figure 3 This is a schematic diagram of the structure of the electronic device in an embodiment of the present invention. Detailed Implementation

[0044] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present application.

[0045] It should be noted that, where there is no conflict, the embodiments and features of the embodiments in this application can be combined with each other. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0046] refer to Figure 1 This embodiment discloses an intelligent resource scheduling method based on dynamic data lineage graphs, which includes the following steps:

[0047] S100: Collects the execution logs and stream metadata of tasks in the computing platform in real time, and constructs a dynamic directed weighted lineage graph based on the execution logs and stream metadata.

[0048] In distributed computing platforms, such as Hadoop or Spark clusters, a sliding time window mechanism is used to capture multi-dimensional feature vectors in real time, including task execution logs, data shard status, and network I / O latency. The execution logs include records of task start time, runtime, input / output data volume, and exception information. Stream metadata includes data dependencies between tasks, data shard locations, and network bandwidth usage. An example of setting the time window length is provided. This means that data collection and map updates are triggered every 5 seconds, and the window length can be dynamically adjusted according to the cluster load.

[0049] Based on the aforementioned multidimensional feature vectors, the dynamic weights of the kinship graph edges are calculated using a weighted combination model. The calculation formula is as follows: ,in, For the source node To the target node The amount of data in the data dependency relationship, For the source node To the target node Transmission delay, For the source node To the target node Dependence strength, , , and This is the adaptive decay coefficient. The adaptive decay coefficient is dynamically adjusted using an online gradient descent algorithm. , , and This is done to minimize historical weight prediction errors. For example, if the actual transmission delay of a certain task link is consistently higher than the predicted value, then... The coefficient is gradually increased (e.g., from 0.5 to 0.7) to increase the weight of the delayed term in subsequent weight calculations.

[0050] In this implementation, nodes represent tasks, and vertices are a vertex in the graph corresponding to each task node. If the target node... Dependence on source node The output data then establishes a connection from... arrive directed edges At this point, the above calculations will be performed. As the current directed edge The initial weights are used to construct a weighted kinship graph. After constructing the weighted kinship graph, the graph topology and weights need to be updated every preset time based on the latest collected metadata. If no new data flows into a task within the window, it is marked as an invalid node and deleted.

[0051] Therefore, by collecting task execution logs and streaming data from the computing platform in real time and constructing a dynamic directed weighted lineage graph, high-precision modeling of complex dependencies between tasks is achieved. This solves the problems of traditional static lineage analysis being unable to adapt to dynamic load changes and resource scheduling lag caused by the simplistic task dependency weights. Specifically, a dynamic weight formula is constructed based on multi-dimensional feature vectors captured by a sliding time window mechanism. Through online optimization of the adaptive decay coefficient, the weight calculation can reflect the impact of sudden data volume on network bandwidth and enhance the distinguishability of high-latency paths. At the same time, the task dependency strength standardization value is combined to quantify the task correlation tightness, thereby breaking through the limitations of the static weight allocation of the traditional PageRank algorithm and enabling the lineage graph weights to dynamically adapt to cluster load fluctuations.

[0052] S200: Calculate the kinship influence coefficient of each node in the dynamic directed weighted kinship graph using a weighted propagation model of forward and reverse dependency paths.

[0053] First, the impact value of forward propagation is calculated based on the forward dependency path. Specifically, the set of downstream nodes of a node is determined for any node in the lineage graph. its downstream node set Defined as: ,in, For the edge set of the kinship graph, Indicates from node To the node The directed edge.

[0054] Based on the calculation in step S100 With the preset level coefficient Calculate the forward propagation effect: .

[0055] Secondly, the impact value of backpropagation is calculated based on the reverse dependency path. Specifically, the set of upstream nodes of a node is determined, for any node in the lineage graph. Its upstream node set Defined as: ,in, Indicates from node To the node The directed edge.

[0056] Based on the calculation in step S100 With the preset level coefficient Calculate the backpropagation effect: ,in, This is the preset exponential decay factor.

[0057] Set the backpropagation gain coefficient This is used to adjust the contribution ratio of back propagation to the kinship influence coefficient. The kinship influence coefficient is calculated as follows: ,Right now .

[0058] S300: Obtain the task to be scheduled and calculate a comprehensive priority score based on the dependency weights of its dynamic lineage graph, the real-time load status of the target node, and the scheduling execution delay.

[0059] Obtain the input parameters for the task to be scheduled: lineage influence coefficient The current load ratio of the target node Scheduling execution delay and task flexibility coefficient .

[0060] It should be noted here that in step S200, the following steps are used: The kinship influence coefficient represents the number of nodes in the kinship diagram. The bloodline influence coefficient is used to quantify the impact of this node on the global bloodline graph, where... This is a general index symbol representing a node. In this step, Indicates tasks to be scheduled The influence coefficient of bloodline, This serves as the task identifier. In the pedigree graph, each task corresponds to a node, meaning... for A special case, namely the task The bloodline influence coefficient is calculated using the same formula for both.

[0061] The bloodline weight is dynamically updated using a preset weight adjustment model. Load weight and elastic weight Specifically, a weight adjustment differential equation is defined to achieve weight adjustment, and the formula is as follows: ,in, and The weight allocation is based on preset initial weight values. The diffusion coefficient is... For comprehensive priority score The second-order space reciprocal is used to reflect the propagation effect of weights between tasks. The preset oscillation range, It is a period of oscillation. Let be the execution time of the current task. Then the formula for calculating the updated weight is: ,in, For weighted components time step The updated value after that, This represents the weight component at the current moment, and the initial weight value at the beginning.

[0062] Based on the weight values ​​calculated above, the comprehensive priority score is calculated using the following formula: ,in, This is the time delay decay factor.

[0063] Therefore, by calculating the comprehensive priority score through a multi-objective optimization function with dynamic weight allocation, and combining the lineage influence coefficient, target node load ratio, and scheduling execution latency, adaptive matching of resource scheduling priority and cluster state is achieved, effectively solving the resource skew problem caused by traditional static weight allocation. This method introduces a dynamic weight adjustment model, utilizing a diffusion term and a periodic oscillation term to collaboratively optimize the weight distribution. The diffusion term smooths the weight differences between tasks through the spatial second derivative of the comprehensive priority score, while the periodic oscillation term dynamically adjusts the intensity of weight fluctuations based on task deadlines, ensuring stronger resource guarantees for high-priority tasks at critical moments. Simultaneously, the nonlinear fusion mechanism employed in the multi-objective optimization function significantly improves the adaptability of the scheduling strategy to sudden load scenarios. For example, when the target node load suddenly increases, the priority score of high-lineage-influence tasks automatically increases due to dynamic weight adjustment, while the comprehensive score of low-elasticity tasks decreases synchronously, thus creating differentiated guarantees during the resource allocation phase.

[0064] S400: Based on the comprehensive priority score, prioritize the tasks to be scheduled and allocate cluster resources according to the ranking results.

[0065] Obtain the comprehensive priority score of the tasks to be scheduled calculated in step S300, sort them in descending order, and generate a priority queue. ,in This is the highest priority task. Based on the total cluster resources... And task priority queues, dynamically set resource allocation thresholds, including: when the overall priority score is greater than or equal to a preset score judgment threshold. At that time, set a high-priority resource threshold. When the overall priority score is greater than or equal to the preset score judgment threshold and less than At that time, set the threshold for medium-optimal resources. When the overall priority score is less than Low-priority resource threshold .

[0066] Resources are allocated to tasks sequentially according to the priority queue: when Set the task as a high-priority task and allocate resources to it. ,in, For nodes The overall priority score, The total score for all tasks in the queue, when Set the task as a medium-optimal task and allocate resources to it. ,when Allocate resources for low-priority tasks ,in, The total amount of resources allocated to high-priority and medium-priority tasks.

[0067] Step S400 achieves significant technological advancements in task ranking and resource allocation based on comprehensive priority scoring through a multi-objective optimization function with dynamic weight allocation and an adaptive mechanism for resource allocation thresholds. Firstly, the comprehensive priority scoring function overcomes the limitations of traditional linear weighted models in adapting to high-load scenarios by nonlinearly coupling the lineage influence coefficient, load normalization term, and exponential decay delay term. For example, when the target node load L... n When the coefficient is 0.8, the high kinship influence coefficient for tasks (such as I) p =125.72) The priority score is automatically reduced by about 17.6% due to the load normalization term, while in low-load scenarios (such as L) n =0.3) Its score improved by 32.4%, achieving dynamic matching between resource allocation and load status; secondly, the dynamic weight allocation model uses diffusion terms With periodic oscillation term The synergistic effect, driven by the aforementioned differential equations, enables the weights to respond in real time to changes in the task queue. Furthermore, the resource allocation threshold division and recycling mechanism, through the dynamic setting of three levels of thresholds (high priority, medium priority, and low priority), ensures that high priority tasks are allocated proportionally based on priority queue sorting and resource constraints, while avoiding waste of cluster resources through the low priority task resource recycling mechanism. In addition, the resource allocation, lineage influence coefficient calculation, and migration decision mechanism form a closed-loop control, significantly improving system stability.

[0068] S500: Predicts the load trend of the target node. When the predicted load of the target node exceeds a threshold and the weight ratio of its critical lineage tasks exceeds a preset threshold, a migration decision is triggered. Critical lineage tasks are those with a lineage influence coefficient greater than a preset threshold.

[0069] In this embodiment, the specific method includes predicting the future load of the target node based on a hybrid model that integrates a deep learning model and a statistical model. The prediction function is: ,in, The privacy-weighted payload prediction value output by the deep learning model. For dynamic mixing coefficients, For constant terms, and The order of the statistical model. and The autoregressive and moving average coefficients of the mixture model are given. Historical load values This represents historical residuals.

[0070] The predicted load and key lineage task weight ratio of the target node are detected. The formula for calculating the key lineage task weight ratio is as follows: ,in A collection of key bloodline missions. For the target node The current set of tasks being carried out For the target node The total weight of key bloodline missions For the target node The total weight of all tasks.

[0071] If the predicted load of the target node exceeds a preset threshold and the weight ratio of the key lineage task exceeds a preset threshold, then a migration task and a target migration node are selected based on the migration utility function and the benefit amplification mechanism. This embodiment defines a migration utility function: ,in, The coefficient representing the influence of bloodline at the node. Reduce the load on the target node. For migration costs, For data transmission delay, For bandwidth sensitivity coefficient, For available bandwidth increments, This represents the total bandwidth of the cluster. This implementation further introduces a benefit amplification factor. ,in, This is the exponential gain factor, used to control the amplification factor of migration gains. In this embodiment, its value range is [value range missing]. ; Task slack time represents the remaining schedulable time for the task. This represents the task execution time, reflecting the time required to complete the task.

[0072] The migration path is dynamically optimized through tree search, and the utility value is calculated based on the migration utility function and the benefit method coefficient. Based on the calculation results, the migration scheme with the highest utility value is selected.

[0073] This technical solution, through the synergistic mechanism of the hybrid load prediction model and the migration utility function, achieves significant technological advancements in predicting the load trend of the target node and triggering migration decisions. First, the load prediction function built based on the hybrid model combines nonlinear fluctuation characteristics with linear trend modeling through online optimization of dynamic hybrid coefficients, reducing prediction errors compared to a single model and ensuring the pre-emptiveness and accuracy of migration decisions. Second, the inclusion of a key lineage task weight constraint in the migration triggering conditions, using the lineage influence coefficient to screen high-priority tasks, solves the problem of decreased system stability caused by blindly migrating key tasks in traditional overload migration strategies. For example, in a task chain ABC, if the lineage influence coefficient I of task AB carried by node B is... A =125.72 exceeds threshold I th =120, then its weight accounts for 79.4% (θ=0.75). After the migration is triggered, the task T3 with low lineage impact is migrated first instead of task A, ensuring the continuity of resources for critical tasks. Furthermore, the migration utility function uses the non-linear coupling between the bandwidth sensitivity coefficient and the total cluster bandwidth to dynamically balance the load reduction and bandwidth release effect when calculating the migration benefits. For example, when B t When the available bandwidth is 500 MB, the available bandwidth increment ΔB = 100 MB contributes only a utility value of ln(1.2)≈0.182, avoiding excessive bandwidth resource allocation that could intensify competition for resources among other tasks. In addition, the benefit amplification factor models the sensitivity of task deadlines through an exponential gain factor. When the task slack time is much longer than the execution time, the migration utility can be effectively amplified, ensuring that high-priority tasks have priority in migration. Meanwhile, the benefit amplification factor of tasks nearing their deadlines will only be slightly amplified, effectively suppressing the risk of blind migration.

[0074] S600: A dynamic feedback control mechanism is formed by periodically updating the kinship map, adjusting the parameters for calculating the influence coefficient, optimizing the parameters for the time series prediction model, and evaluating the migration strategy.

[0075] Graph updates are triggered based on a preset period or graph update threshold. For dynamic adjustment of edge weights, the adaptive decay coefficient is optimized using an online gradient descent algorithm, based on the dynamic weights calculated in step S100 and combined with the latest collected execution logs and stream metadata. Based on the forward / backward propagation model in step S200, the lineage influence coefficient is recalculated and the set of key lineage tasks is updated. For example, in a task chain ABC, if the data volume of downstream task C of task B suddenly increases, the edge weight W is dynamically adjusted. BC And recalculate the bloodline influence coefficient I for Task B. B , if I B Exceeding threshold I thIf so, then it will be included in the key bloodline task set K.

[0076] Online optimization is performed on the gain coefficient and attenuation factor to dynamically adjust the backpropagation gain coefficient and exponential attenuation factor through an online gradient descent algorithm, minimizing the prediction error of the historical lineage influence coefficient; adaptive updates are performed on the task level coefficient, dynamically adjusting the task level coefficient based on the actual performance of the task, such as the task elasticity coefficient or resource utilization ratio.

[0077] The mixing coefficients of the hybrid model are dynamically updated based on load prediction and adjusted dynamically using an online gradient descent algorithm to minimize prediction error.

[0078] The migration utility function parameters are calibrated by dynamically adjusting them based on the task delay and resource recovery efficiency after migration. At the same time, the migration trigger threshold is adjusted based on the change in the weight ratio of key lineage tasks after migration. For example, if the resource guarantee for key personnel is insufficient after migration, the threshold is lowered to expand the coverage of key tasks.

[0079] In summary, the intelligent resource scheduling method based on dynamic data lineage graph disclosed in this embodiment achieves a synergistic improvement in resource scheduling efficiency and fairness in a distributed computing platform through a closed-loop optimization of the entire chain, including dynamic data lineage graph construction, lineage influence coefficient propagation model, comprehensive priority scoring, resource allocation strategy, and migration decision. First, a dynamic directed weighted lineage graph is constructed based on task execution logs and stream metadata. By real-time collection of multi-dimensional feature vectors (including data volume, transmission latency, and task dependency strength), and combined with a dynamic weight allocation model, the latency sensitivity, scale effect, and task relevance of data dependencies are quantified, overcoming the limitations of traditional static lineage analysis and enabling weight calculation to respond in real-time to sudden changes in task dependencies. Second, the lineage influence coefficient is calculated through a weighted propagation model of forward and reverse dependency paths, comprehensively considering the impact of tasks on downstream nodes and the constraints on upstream nodes, and introducing tree search to optimize the propagation path, effectively improving the accuracy of the propagation path compared to the traditional PageRank algorithm. Furthermore, the comprehensive priority scoring module, through dynamic weighting... The redistribution mechanism and nonlinear utility function enable adaptive adjustment of task priorities, solving the resource skew problem of fixed-weight schemes during load surges. The resource allocation module uses a three-level threshold division of high / medium / low priority and a low-priority task resource recovery mechanism to ensure the upper limit of high-priority tasks' resources while preventing low-priority tasks from being completely deprived due to resource competition. The migration decision module adopts a hybrid load prediction method integrating deep learning models and statistical models. By dynamically mixing coefficients to balance nonlinear fluctuation characteristics and linear trend modeling, the prediction error is significantly reduced compared to a single model. Combined with a dual-condition triggering mechanism (predicted load exceeding the limit and the weight ratio of critical lineage tasks exceeding the limit), it prioritizes the migration of low-lineage tasks to avoid interruption of high-lineage tasks. Finally, through a four-dimensional closed-loop system formed by periodically updating the lineage graph topology, online parameter optimization, time-series prediction model parameter calibration, and migration strategy evaluation, the system can maintain scheduling flexibility and critical task guarantee rate under sudden load scenarios. This overcomes the shortcomings of existing technologies in responding to dynamic loads and rigid resource allocation, ultimately achieving a dynamic balance between resource scheduling efficiency and fairness.

[0080] refer to Figure 2 This embodiment further discloses an intelligent resource scheduling system based on dynamic data lineage graphs, including:

[0081] Data acquisition module 21 is used to collect execution logs and streaming metadata of tasks in the computing platform in real time, and construct a dynamic directed weighted lineage graph based on the execution logs and streaming metadata; including: capturing the execution logs and streaming metadata of tasks in real time through a preset time window mechanism; extracting multi-dimensional feature vectors based on the execution logs and streaming metadata, wherein the multi-dimensional feature vectors include at least data volume, data transmission latency, and task dependency strength; and calculating the dynamic weights of the lineage graph edges based on the multi-dimensional feature vectors through a weighted combination model, the calculation formula being: ,in, For the source node To the target node The amount of data in the data dependency relationship, For the source node To the target node Transmission delay, For the source node To the target node Dependence strength, , , and The adaptive decay coefficient is used; with nodes as vertices of the lineage graph and data dependencies as directed edges, a weighted lineage graph is formed through the dynamic weights.

[0082] The kinship influence coefficient calculation module 22 is used to calculate the kinship influence coefficient of each node in the dynamic directed weighted kinship graph using a weighted propagation model of forward dependency paths and reverse dependency paths; including: for nodes All downstream nodes Based on edge weights in the kinship graph With preset task level coefficient The sum of the products is used to calculate the forward propagation impact value; for nodes... All upstream nodes Based on edge weights in the kinship graph With preset task level coefficient The backpropagation effect is calculated by summing the power functions of the products; the backpropagation gain coefficient is preset. and exponential decay factor The forward propagation influence value and the backward propagation influence value are weighted and combined to obtain the node lineage influence coefficient. The calculation formula is: ,in For nodes The set of downstream nodes, For nodes The set of upstream nodes, and This is the preset task level coefficient.

[0083] The comprehensive priority score calculation module 23 is used to obtain the task to be scheduled and calculate the comprehensive priority score based on the dependency weights of its dynamic lineage graph, the real-time load status of the target node, and the scheduling execution latency; including: obtaining the lineage influence coefficient of the task to be scheduled. The current load ratio of the target node Scheduling execution delay and task flexibility coefficient Parameters; the bloodline weights are dynamically updated through a preset weight adjustment model. Load weight and elastic weight The parameters and weights are fused using a multi-objective optimization function to calculate the comprehensive priority score. The calculation formula is as follows: ,in, This is the time delay decay factor.

[0084] Resource allocation module 24 is used to prioritize the tasks to be scheduled based on the comprehensive priority score, and allocate cluster resources according to the ranking result; including: sorting the comprehensive priority scores of the tasks to be scheduled in descending order to generate a priority queue. ,in As the highest priority task; based on the total cluster resources And task priority queues, dynamically set resource allocation thresholds, including: when the overall priority score is greater than or equal to a preset score judgment threshold. At that time, set a high-priority resource threshold. When the overall priority score is greater than or equal to the preset score judgment threshold and less than At that time, set the threshold for medium-optimal resources. When the overall priority score is less than Low-priority resource threshold Resources are allocated to tasks sequentially according to the priority queue: when Set the task as a high-priority task and allocate resources to it. ,in, For nodes The overall priority score, The total score for all tasks in the queue, when Set the task as a medium-optimal task and allocate resources to it. ,when Allocate resources for low-priority tasks ,in, The total amount of resources allocated to high-priority and medium-priority tasks.

[0085] The load migration module 25 is used to predict the load trend of the target node. When it detects that the predicted load of the target node exceeds a threshold and the weight ratio of the key lineage tasks it carries exceeds a preset threshold, a migration decision is triggered. This includes: predicting the future load of the target node based on a hybrid model that integrates a deep learning model and a statistical model. The prediction function is: ,in, The privacy-weighted payload prediction value output by the deep learning model. For dynamic mixing coefficients, For constant terms, and The order of the statistical model. and The autoregressive and moving average coefficients of the mixture model are given. Historical load values This refers to historical residual terms; the predicted load and key lineage task weight ratio of the detection target node are calculated using the following formula: ,in A collection of key bloodline missions. For the target node The current set of tasks being carried out For the target node The total weight of key bloodline missions For the target node The total weight of all tasks; if the predicted load of the target node exceeds a preset threshold and the weight ratio of the key lineage tasks exceeds a preset threshold, then the migration task and target migration node are selected based on the migration utility function and the benefit amplification mechanism.

[0086] Figure 3 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as... Figure 3 As shown, the electronic device 50 includes: a processor 501, a memory 502, and a bus 503;

[0087] The processor 501 and the memory 502 communicate with each other via the bus 503; the processor 501 is used to call the program instructions in the memory 502 to execute the methods provided in the above-described embodiments.

[0088] This embodiment provides a non-transitory computer-readable storage medium that stores computer instructions that cause a computer to execute the methods provided in the above-described embodiments.

[0089] Those skilled in the art will understand that all or part of the steps of the above-described method implementation can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above-described method implementation. The aforementioned storage medium includes various storage media capable of storing program code, such as ROM, RAM, magnetic disk, or optical disk.

[0090] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0091] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of each embodiment or some parts of the embodiments.

[0092] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An intelligent resource scheduling method based on a dynamic data bloodline map, characterized in that, The method comprises: real-time collection of execution logs and flow metadata of tasks in a computing platform, and construction of a dynamic directed weighted bloodline graph based on the execution logs and flow metadata; calculation of bloodline influence coefficients of each node in the dynamic directed weighted bloodline graph through a weighted propagation model of forward and reverse dependency paths; Obtain the tasks to be scheduled and calculate a comprehensive priority score based on their dependency weights in the dynamic lineage graph, the real-time load status of the target nodes, and the scheduling execution latency, including: obtaining the lineage influence coefficient of the tasks to be scheduled. The current load ratio of the target node Scheduling execution delay and task flexibility coefficient Parameters; the bloodline weights are dynamically updated through a preset weight adjustment model. Load weight and elastic weight The parameters and weights are fused using a multi-objective optimization function to calculate the comprehensive priority score. The calculation formula is as follows: ,in, This is the time delay attenuation factor; priority sorting of tasks to be scheduled based on the comprehensive priority score, and allocation of cluster resources according to the sorting result; prediction of the load trend of the target node, and triggering of a migration decision when it is detected that the predicted load of the target node exceeds a threshold and the weight proportion of the key bloodline tasks borne by the target node exceeds a preset threshold. 2.The method of claim 1, wherein, The real-time collection of execution logs and flow metadata of tasks in a computing platform, and construction of a dynamic directed weighted bloodline graph based on the execution logs and flow metadata comprises: real-time capture of execution logs and flow metadata of tasks through a preset time window mechanism, extraction of a multi-dimensional feature vector based on the execution logs and flow metadata, the multi-dimensional feature vector at least including data volume, data transmission delay and task dependency strength; A dynamic weight of a blood relation graph edge is calculated by a weighted combination model based on the multi-dimensional feature vector, and a calculation formula is as follows: wherein, is a data amount in a data dependency relationship from a source node to a target node , is a transmission delay from the source node to the target node , is a dependency strength from the source node to the target node , , , and are adaptive attenuation coefficients. forming a weighted bloodline graph through the dynamic weights, with nodes as vertices of the bloodline graph and data dependency relationships as directed edges. 3.The method of claim 1, wherein, The calculation of bloodline influence coefficients of each node in the dynamic directed weighted bloodline graph through a weighted propagation model of forward and reverse dependency paths comprises: For nodes All downstream nodes Based on edge weights in the kinship graph With preset task level coefficient The sum of the products is used to calculate the forward propagation effect value; For nodes All upstream nodes Based on edge weights in the kinship graph With preset task level coefficient The sum of the power functions of the product is used to calculate the backpropagation effect value; Through the preset backpropagation gain coefficient and exponential decay factor The forward propagation influence value and the backward propagation influence value are weighted and combined to obtain the node lineage influence coefficient. The calculation formula is: ,in For nodes The set of downstream nodes, For nodes The set of upstream nodes, and This is the preset task level coefficient. 4.The method of claim 1, wherein, The priority sorting of tasks to be scheduled based on the comprehensive priority score, and the allocation of cluster resources according to the sorting result comprises: ordering the composite priority scores of the tasks to be scheduled in descending order to generate a priority queue wherein is the highest priority task; According to the total amount of cluster resources and the task priority queue, a resource allocation threshold is dynamically set, including: when the comprehensive priority score is greater than or equal to a preset score judgment threshold , a high-priority resource threshold is set , when the comprehensive priority score is greater than or equal to the preset score judgment threshold and less than , a medium-priority resource threshold is set , and when the comprehensive priority score is less than , a low-priority resource threshold is set . According to the order of the priority queue, resources are assigned to tasks in turn: when a task is set as a high-priority task and resources are assigned to the high-priority task wherein is the combined priority score of the node , is the total score of all tasks in the queue, when a task is set as a medium-priority task and resources are assigned to the medium-priority task when resources are assigned to low-priority tasks wherein is the sum of the resources already assigned to high-priority and medium-priority tasks. 5.The method of claim 1, wherein, The prediction of the load trend of the target node, and the triggering of a migration decision when it is detected that the predicted load of the target node exceeds a threshold and the weight proportion of the key bloodline tasks borne by the target node exceeds a preset threshold comprises: Hybrid model based on integrated deep learning model and statistical model for predicting future load of target node , the prediction function is: , wherein, is the privacy-weighted load prediction value output by the deep learning model, is a dynamic mixing coefficient, is a constant term, and is the order of the statistical model, and are the autoregressive and moving average coefficients of the hybrid model, is a historical load value, is a historical residual term; The predicted load of the target node and a key bloodline task weight proportion are detected, and a calculation formula of the key bloodline task weight proportion is: wherein a key bloodline task set, a target node a task set currently borne by the target node, a target node a total weight of the key bloodline task, a target node a total weight of all tasks; If the predicted load of the target node exceeds a preset threshold and the weight proportion of the key bloodline tasks exceeds a preset threshold, selecting a migration task and a target migration node based on a migration utility function and a revenue amplification mechanism.

6. An intelligent resource scheduling system based on dynamic data pedigree map, characterized in that, It comprises: a data collection module configured to collect execution logs and flow metadata of tasks in a computing platform in real time, and construct a dynamic directed weighted bloodline graph based on the execution logs and flow metadata; a bloodline influence coefficient calculation module configured to calculate bloodline influence coefficients of each node in the dynamic directed weighted bloodline graph through a weighted propagation model of forward and reverse dependency paths; The comprehensive priority score calculation module is used to obtain the tasks to be scheduled and calculate a comprehensive priority score based on the dependency weights of their dynamic lineage graph, the real-time load status of the target nodes, and the scheduling execution latency. This includes: obtaining the lineage influence coefficient of the tasks to be scheduled. The current load ratio of the target node Scheduling execution delay and task flexibility coefficient Parameters; the bloodline weights are dynamically updated through a preset weight adjustment model. Load weight and elastic weight The parameters and weights are fused using a multi-objective optimization function to calculate the comprehensive priority score. The calculation formula is as follows: ,in, This is the time delay attenuation factor; a resource allocation module configured to sort tasks to be scheduled based on the comprehensive priority score, and allocate cluster resources according to the sorting result; a load migration module configured to predict the load trend of the target node, and trigger a migration decision when it is detected that the predicted load of the target node exceeds a threshold and the weight proportion of the key bloodline tasks borne by the target node exceeds a preset threshold.

7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and the computer program is executed by the processor to implement the method of any one of claims 1-5.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the method of any one of claims 1-5.

9. A computer program product comprising a computer program, characterized in that, The computer program, when executed by the processor, implements the method of any one of claims 1-5.

Images