Intelligent resource scheduling method and system based on dynamic data consanguinity map

Abstract

The invention discloses an intelligent resource scheduling method and system based on a dynamic data consanguinity atlas, and relates to the technical field of resource scheduling, the method comprises the following steps: collecting execution logs and flow metadata of tasks in a computing platform in real time, and constructing a dynamic directed weighted consanguinity atlas; calculating a blood relationship influence coefficient of each node in the dynamic directed weighted blood relationship map; obtaining a to-be-scheduled task, and calculating a comprehensive priority score based on the dependency weight of the dynamic blood relationship map of the to-be-scheduled task, the real-time load state of the target node and the scheduling execution time delay; performing priority ranking on the to-be-scheduled tasks based on the comprehensive priority score, and allocating cluster resources; and predicting the load trend of the target node, and triggering a migration decision when detecting that the predicted load of the target node exceeds a threshold value and the weight ratio of the key consanguinity tasks borne by the target node exceeds a preset threshold value. Through a dynamic consanguinity map and an intelligent scheduling algorithm, high efficiency and fairness of resource allocation are realized, the cluster utilization rate is improved, and task delay is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of resource scheduling, and in particular to an intelligent resource scheduling method and system based on a dynamic data lineage graph. Background Art

[0002] Currently, in a distributed computing environment, traditional resource scheduling methods generally have the following technical defects: First, existing scheduling strategies usually make decisions only based on static task attributes (such as preset priorities or resource requirements), and fail to fully consider the complex data dependencies between tasks, resulting in resource allocation that cannot reflect the actual impact of tasks on the global lineage graph; Second, the prediction accuracy of dynamic load changes and task execution delays is insufficient, making it difficult to actively trigger resource rebalancing before load overload, resulting in low utilization of cluster resources or a sharp increase in task delays; Third, the migration decision mechanism lacks a quantitative assessment of task lineage weights, and may incorrectly migrate key lineage tasks in the case of load overload, leading to a decline in system stability. For example, some solutions calculate task importance through the PageRank algorithm, but its static graph modeling cannot adapt to the real-time changes of task dependencies, and the lack of a backpropagation mechanism leads to one-sided evaluation of lineage impacts; Other solutions use a multi-objective optimization function with fixed weights for priority scoring, but the weight parameters cannot be dynamically adjusted according to the cluster state, resulting in unbalanced resource allocation in high-load scenarios. In addition, traditional migration strategies rely on a single trigger condition of the load threshold, without combining the analysis of the proportion of task lineage weights, and are prone to missing the opportunity to guarantee key task resources due to blindly migrating tasks with low lineage impacts. These problems together make it difficult for the existing technology to achieve the collaborative optimization of resource scheduling efficiency and fairness in a dynamic load environment. Summary of the Invention

[0003] In view of this, the present invention proposes an intelligent resource scheduling method and system based on a dynamic data lineage graph, which can improve the utilization rate of cluster resources and the reliability of task scheduling. The present invention provides the following technical solutions: An intelligent resource scheduling method based on a dynamic data lineage graph, the method includes: Real-time collecting the execution logs and flow metadata of tasks in a computing platform, and constructing a dynamic directed weighted lineage graph based on the execution logs and flow metadata; Calculating the lineage influence coefficient of each node in the dynamic directed weighted lineage graph through a weighted propagation model of forward and reverse dependency paths; Obtaining a task to be scheduled and calculating a comprehensive priority score based on the dependency weight of its dynamic lineage graph, the real-time load status of the target node, and the scheduling execution delay; Performing priority sorting on the tasks to be scheduled based on the comprehensive priority score, and allocating cluster resources according to the sorting result; Predict the load trend of the target node. When it is detected that the predicted load of the target node exceeds the threshold and the proportion of the weight of the critical lineage tasks it bears exceeds the preset threshold, trigger a migration decision.

[0004] Optionally, the execution logs and flow metadata of tasks in the real-time acquisition and calculation platform are collected, and constructing a dynamic directed weighted lineage graph based on the execution logs and flow metadata includes: Real-time capture the execution logs and flow metadata of tasks through a preset time window mechanism, extract multi-dimensional feature vectors based on the execution logs and flow metadata, and the multi-dimensional feature vectors at least include the data volume, data transmission delay, and task dependency strength; Calculate the dynamic weight of the edges of the lineage graph based on the multi-dimensional feature vectors through a weighted combination model, and the calculation formula is: , where, is the source node to the target node the data volume in the data dependency relationship, is the source node to the target node the transmission delay, is the source node to the target node the dependency strength, , , and are adaptive attenuation coefficients; Taking nodes as the vertices of the lineage graph and data dependency relationships as directed edges, form a weighted lineage graph through the dynamic weights.

[0005] Optionally, calculating the lineage influence coefficient of each node in the dynamic directed weighted lineage graph through a weighted propagation model of forward and reverse dependency paths includes: For all downstream nodes of node , calculate the forward propagation influence value based on the sum of the product of the edge weight in the lineage graph and the preset task level coefficient ; For all upstream nodes of node , calculate the reverse propagation influence value based on the sum of the power functions of the product of the edge weight in the lineage graph and the preset task level coefficient ; Through a preset reverse propagation gain coefficient and an exponential decay factor , perform a weighted combination on the forward propagation influence value and the reverse propagation influence value to obtain the node lineage influence coefficient , the calculation formula is: , where is the set of downstream nodes of node , is the set of upstream nodes of node , and are preset task level coefficients.

[0006] Optionally, the obtaining of the task to be scheduled and the calculation of the comprehensive priority score based on the dependency weight of its dynamic lineage graph, the real-time load status of the target node, and the scheduling execution delay include: Obtaining the lineage influence coefficient of the task to be scheduled , the current load ratio of the target node , the scheduling execution delay and the task elasticity coefficient parameters; Dynamically updating the lineage weight , the load weight and the elasticity weight through a preset weight adjustment model; Fusing the parameters and weights through a multi-objective optimization function to calculate the comprehensive priority score, and the calculation formula is: , where is the delay attenuation factor.

[0007] Optionally, the prioritizing of the tasks to be scheduled based on the comprehensive priority score and the allocation of cluster resources according to the sorting result include: Sorting the comprehensive priority scores of the tasks to be scheduled in descending order to generate a priority queue , where is the highest priority task; According to the total amount of cluster resources and the task priority queue, dynamically setting the resource allocation threshold, including: when the comprehensive priority score is greater than or equal to the preset score judgment threshold , setting the high-priority resource threshold , when the comprehensive priority score is greater than or equal to the preset score judgment threshold and less than , setting the medium-priority resource threshold When the comprehensive priority score is less than , the low-priority resource threshold ; Allocating resources to tasks in sequence according to the order of the priority queue: when , setting the task as a high-priority task and allocating resources to the high-priority task , where is the node The comprehensive priority score, is the total score of all tasks in the queue. When , set the task as a medium-priority task and allocate resources to the medium-priority task , when , allocate resources to the low-priority task , where is the sum of the resources already allocated to the high-priority and medium-priority tasks.

[0008] Optionally, when predicting the load trend of the target node, when it is detected that the predicted load of the target node exceeds the threshold and the weight ratio of the critical lineage tasks it bears exceeds the preset threshold, the migration decision triggered includes: Predict the future load of the target node based on a hybrid model of an integrated deep learning model and a statistical model , and the prediction function is: , where is the privacy-weighted load prediction value output by the deep learning model, is the dynamic mixing coefficient, is the constant term, and is the order of the statistical model, and are the autoregressive and moving average coefficients of the hybrid model, is the historical load value, is the historical residual term; Detect the predicted load of the target node and the weight ratio of the critical lineage tasks. The calculation formula for the weight ratio of the critical lineage tasks is: , where is the set of critical lineage tasks, is the target node The set of tasks currently carried, is the target node The total weight of the critical lineage tasks on, is the target node The total weight of all tasks; If the predicted load of the target node exceeds the preset threshold and the weight ratio of the critical lineage tasks exceeds the preset threshold, then select the migration task and the target migration node based on the migration utility function and the benefit amplification mechanism.

[0009] The present invention further discloses an intelligent resource scheduling system based on a dynamic data lineage graph, including: A data collection module for real-time collecting the execution logs and stream metadata of tasks in the computing platform, and constructing a dynamic directed weighted lineage graph based on the execution logs and stream metadata; A blood relationship influence coefficient calculation module, configured to calculate the blood relationship influence coefficient of each node in the dynamic directed weighted blood relationship graph through a weighted propagation model of forward and reverse dependence paths; A comprehensive priority score calculation module, configured to obtain a task to be scheduled and calculate a comprehensive priority score based on the dependence weight of its dynamic blood relationship graph, the real-time load status of the target node, and the scheduling execution delay; A resource allocation module, configured to sort the tasks to be scheduled based on the comprehensive priority score and allocate cluster resources according to the sorting result; A load migration module, configured to predict the load trend of the target node, and trigger a migration decision when it is detected that the predicted load of the target node exceeds a threshold and the weight ratio of the critical blood relationship tasks carried by it exceeds a preset threshold.

[0010] The present invention further discloses a computer-readable storage medium, where the storage medium stores a computer program, and when the computer program is executed by a processor, the above method is implemented.

[0011] The present invention further discloses an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the program, the above method is implemented.

[0012] The present invention further discloses a computer program product, including a computer program, and when the computer program is executed by a processor, the above method is implemented.

[0013] According to the technical solution of the present invention, a dynamic directed weighted blood relationship graph is constructed by collecting task execution logs and stream metadata in real time, and the node blood relationship influence coefficient is calculated by combining the weighted propagation models of forward and reverse dependence paths, breaking through the limitations of traditional static blood relationship analysis and one-way propagation, enabling the task importance assessment to dynamically reflect the real-time changes of data dependence relationships. This method constructs a comprehensive priority score system for dynamic weight allocation through multi-dimensional parameter fusion, realizes the adaptive matching of resource scheduling priorities and cluster states, solves the problem of resource skew caused by traditional fixed weight allocation, and significantly improves the scheduling flexibility in high-load scenarios. Further, through a dual-condition trigger mechanism of load trend prediction and the weight ratio of critical blood relationship tasks, combined with an optimization algorithm of a migration utility function and a benefit amplification coefficient, when it is detected that the predicted load of the target node exceeds the threshold and the weight ratio of the critical blood relationship tasks carried by it exceeds the limit, low blood relationship influence tasks are preferentially migrated and the migration path is dynamically selected, effectively avoiding the destruction of critical task resource guarantee caused by blind migration, thereby improving the cluster resource utilization rate while reducing task latency, and realizing the coordinated optimization of resource scheduling efficiency and fairness. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] For purposes of illustration and not limitation, the present invention will now be described in connection with embodiments and drawings of the present invention, wherein: Figure 1 is a schematic flowchart of an intelligent resource scheduling method based on a dynamic data lineage graph in an embodiment of the present invention; Figure 2 is a schematic flowchart of an intelligent resource scheduling system based on a dynamic data lineage graph in an embodiment of the present invention; Figure 3 is a schematic structural diagram of an electronic device in an embodiment of the present invention. Detailed Embodiments

[0015] In order to enable those skilled in the art to better understand the solution of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0016] It should be noted that, without conflict, the embodiments of the present application and the features in the embodiments may be combined with each other. The embodiments of the present application will be described in detail below with reference to the drawings.

[0017] Referring to Figure 1 , this embodiment discloses an intelligent resource scheduling method based on a dynamic data lineage graph, and the method includes the following steps: S100: Real-time collect the execution logs and stream metadata of tasks in the computing platform, and construct a dynamic directed weighted lineage graph based on the execution logs and stream metadata.

[0018] In a distributed computing platform, such as a Hadoop or Spark cluster, multi-dimensional feature vectors such as task execution logs, data shard status, and network I / O latency are captured in real time through a sliding time window mechanism, where: the execution logs include recording task start time, running duration, input / output data volume, exception information, etc.; the stream metadata includes data dependencies between tasks, data shard locations, network transmission bandwidth occupancy, etc.; the time window is set with an exemplary window length , that is, data collection and graph update are triggered every 5 seconds, and the window length can be dynamically adjusted according to the cluster load.

[0019] Calculate the dynamic weights of the edges of the lineage graph through a weighted combination model based on the above multi-dimensional feature vectors, and the calculation formula is: , where is the source node to the target node The amount of data in the data dependency relationship is the source node to the target node transmission delay is the source node to the target node dependency strength , , and are the adaptive attenuation coefficients. For the adaptive attenuation coefficients, they are dynamically adjusted through the online gradient descent algorithm , , and , to minimize the historical weight prediction error. For example, if the actual transmission delay of a certain task link is higher than the predicted value for a long time, then the coefficient gradually increases (such as from 0.5 to 0.7) to increase the weight of the delay term in the subsequent weight calculation.

[0020] In this embodiment, the nodes represent tasks, and the vertices are the vertices in the graph corresponding to each task node. If the target node depends on the output data of the source node , then a directed edge is established from to , and at this time, the above calculated is used as the initial weight of the current directed edge , thus forming a weighted lineage graph. After constructing the weighted lineage graph, it is also necessary to update the graph topology and weights based on the latest collected metadata every preset time. If a certain task has no new data flowing in within the window, it is marked as an invalid node and deleted.

[0021] It can be seen that by collecting the task execution logs and flow metadata of the computing platform in real time and constructing a dynamic directed weighted lineage graph, high-precision modeling of the complex dependency relationships between tasks is achieved, and the problems of traditional static lineage analysis being unable to adapt to dynamic load changes and resource scheduling lags caused by the simplification of task dependency weights are solved. Specifically, a dynamic weight formula is constructed based on the multi-dimensional feature vectors captured by the sliding time window mechanism. Through the online optimization of the adaptive attenuation coefficient, the weight calculation can not only reflect the impact of burst data volume on network bandwidth but also enhance the discrimination of high-delay paths. At the same time, the tightness of task association is quantified by combining the standardized value of task dependency strength, thus breaking through the limitations of the static weight allocation of the traditional PageRank algorithm and enabling the weights of the lineage graph to dynamically adapt to the load fluctuations of the cluster.

[0022] ​S200: Calculate the blood relationship influence coefficient of each node in the dynamic directed weighted blood relationship graph through the weighted propagation model of the forward dependence path and the reverse dependence path.

[0023] First, calculate the influence value of forward propagation based on the forward dependence path. Specifically, determine the set of downstream nodes of the node. For any node in the blood relationship graph , its set of downstream nodes is defined as: , where is the set of edges of the blood relationship graph, represents the directed edge from node to node .

[0024] Based on the calculated in step S100 and the preset level coefficient calculate the forward propagation influence value: .

[0025] Secondly, calculate the influence value of reverse propagation based on the reverse dependence path. Specifically, determine the set of upstream nodes of the node. For any node in the blood relationship graph , its set of upstream nodes is defined as: , where represents the directed edge from node to node .

[0026] Based on the calculated in step S100 and the preset level coefficient calculate the reverse propagation influence value: , where is the preset exponential decay factor.

[0027] Set the reverse propagation gain coefficient to adjust the contribution ratio of reverse propagation to the blood relationship influence coefficient. Calculate the blood relationship influence coefficient: , that is .

[0028] S300: Obtain the task to be scheduled and calculate the comprehensive priority score based on the dependence weight of its dynamic blood relationship graph, the real-time load status of the target node, and the scheduling execution delay.

[0029] Obtain the input parameters of the task to be scheduled: the blood relationship influence coefficient , the current load ratio of the target node , the scheduling execution delay and the task elasticity coefficient .

[0030] It should be noted here that in step S200, represents the blood relationship influence coefficient, which represents the node in the blood relationship graph 's blood relationship influence coefficient, used to quantify the influence of this node on the global blood relationship graph, where is a general index symbol representing a node. In this step, represents the blood relationship influence coefficient of the task to be scheduled , is the task identifier. In the blood relationship graph, each task corresponds to a node, that is to say, is a special case of, that is, the blood relationship influence coefficient of task , and the calculation formulas of both are the same.

[0031] Dynamically update the blood relationship weight , load weight and elastic weight through a preset weight adjustment model. Specifically, define a weight adjustment differential equation to achieve weight adjustment, and the formula is: , where and , the weight allocation is based on a preset initial weight value, is the diffusion coefficient, is the comprehensive priority score 's second-order spatial reciprocal, used to reflect the propagation effect of the weight among tasks, is the preset oscillation amplitude, is the oscillation period, is the execution time of the current task. Then the calculation formula of the updated weight is: , where is the updated value of the weight component after the time step , is the weight component at the current moment, which represents the initial weight value at the beginning.

[0032] Based on the calculated weight value above, calculate the comprehensive priority score, and the calculation formula is: , where is the delay attenuation factor.

[0033] It can be seen that by calculating the comprehensive priority score through the multi-objective optimization function with dynamic weight allocation, based on the combination of the blood relationship influence coefficient, the target node load ratio, and the scheduling execution delay, the adaptive matching of the resource scheduling priority and the cluster state is realized, effectively solving the resource skew problem caused by traditional static weight allocation. By introducing a dynamic weight adjustment model, this method uses the diffusion term and the periodic oscillation term to jointly optimize the weight distribution. The diffusion term smooths the weight differences between tasks through the spatial second derivative of the comprehensive priority score, and the periodic oscillation term dynamically adjusts the weight fluctuation intensity according to the task deadline, enabling high-priority tasks to obtain stronger resource guarantees at critical moments. At the same time, the non-linear fusion mechanism adopted in the multi-objective optimization function significantly improves the adaptability of the scheduling strategy to burst load scenarios. For example, when the load of the target node suddenly increases, the priority score of tasks with high blood relationship influence automatically increases due to dynamic weight adjustment, while the comprehensive score of low-elastic tasks decreases synchronously, thus forming differential guarantees in the resource allocation stage.

[0034] S400: Perform priority sorting on the tasks to be scheduled based on the comprehensive priority score, and allocate cluster resources according to the sorting result.

[0035] Obtain the comprehensive priority scores of the tasks to be scheduled calculated in step S300, perform a descending sort on them, and generate a priority queue , where is the task with the highest priority. According to the total amount of cluster resources and the task priority queue, dynamically set resource allocation thresholds, including: when the comprehensive priority score is greater than or equal to the preset score judgment threshold , set the high-priority resource threshold , when the comprehensive priority score is greater than or equal to the preset score judgment threshold and less than , set the medium-priority resource threshold When the comprehensive priority score is less than , the low-priority resource threshold .

[0036] Allocate resources to tasks in sequence according to the order of the priority queue: when , set the task as a high-priority task and allocate resources to the high-priority task , where is the comprehensive priority score of node , is the total score of all tasks in the queue. When , set the task as a medium-priority task and allocate resources to the medium-priority task , when , allocate resources to the low-priority task , where The sum of the resources allocated to high-priority tasks and medium-priority tasks.

[0037] Step S400 achieves significant technological progress in task sorting and resource allocation based on the comprehensive priority score through a multi-objective optimization function with dynamic weight allocation and a resource allocation threshold adaptation mechanism. First, the comprehensive priority score function breaks through the adaptation limitations of the traditional linear weighted model for high-load scenarios through the non-linear coupling of the blood relationship influence coefficient, load normalization term, and exponential decay time delay term. For example, when the load L of the target node n = 0.8, the priority score of a task with a high blood relationship influence coefficient (such as I p = 125.72) is automatically reduced by approximately 17.6% due to the load normalization term, while in a low-load scenario (such as L n = 0.3), its score increases by 32.4%, achieving dynamic matching of resource allocation and load status. Second, the dynamic weight allocation model enables the weight to respond in real time to changes in the task queue through the synergistic effect of the diffusion term and the periodic oscillation term under the drive of the above differential equation. Further, the resource allocation threshold division and recycling mechanism, through the dynamic setting of high / medium / low priority three-level thresholds, combines resource constraint conditions on the basis of priority queue sorting, ensuring proportional allocation of high-priority tasks and avoiding waste of cluster resources through the resource recycling mechanism for low-priority tasks. In addition, the resource allocation, blood relationship influence coefficient calculation, and migration decision mechanism form a closed-loop control, significantly improving system stability.

[0038] S500: Predict the load trend of the target node. When it is detected that the predicted load of the target node exceeds the threshold and the proportion of the weight of the critical blood relationship tasks it bears exceeds the preset threshold, trigger a migration decision. Among them, the critical blood relationship tasks are tasks with a blood relationship influence coefficient greater than the preset threshold.

[0039] In this embodiment, specifically, a hybrid model based on an integrated deep learning model and a statistical model is used to predict the future load of the target node , and the prediction function is: , where is the privacy-weighted load prediction value output by the deep learning model, is the dynamic mixing coefficient, is the constant term, and are the orders of the statistical model, and are the autoregressive and moving average coefficients of the hybrid model, is the historical load value, is the historical residual term.

[0040] Detect the predicted load of the target node and the proportion of the weights of key lineage tasks. The formula for calculating the proportion of the weights of key lineage tasks is as follows: where is the set of key lineage tasks, is the target node is the set of tasks currently carried by the target node, is the target node is the total weight of the key lineage tasks on the target node, is the target node is the total weight of all tasks.

[0041] If the predicted load of the target node exceeds the preset threshold and the proportion of the weights of key lineage tasks exceeds the preset threshold, then select the migration tasks and the target migration node based on the migration utility function and the benefit amplification mechanism. In this embodiment, a migration utility function is defined as follows: where, is the node lineage influence coefficient, is the reduction in the load of the target node, is the migration cost, is the data transmission delay, is the bandwidth sensitivity coefficient, is the increment of available bandwidth, is the total bandwidth of the cluster. In this embodiment, a benefit amplification factor is further introduced, where is the exponential gain factor, which is used to control the amplification ratio of the migration benefit. In this embodiment, its value range is ; is the task slack time, which represents the remaining schedulable time of the task; is the task execution time, which is used to reflect the time required to complete the task.

[0042] Dynamically optimize the migration path through tree search, calculate the utility value based on the migration utility function and the benefit method coefficient, and select the migration scheme with the maximum utility value based on the calculation result.

[0043] Through the technical solution of this step, through the collaborative mechanism of the hybrid load prediction model and the migration utility function, significant technological progress has been achieved in the process of predicting the load trend of the target node and triggering the migration decision: First, based on the load prediction function constructed by the hybrid model, through the online optimization of the dynamic mixing coefficient, the non-linear fluctuation characteristics are combined with the linear trend modeling, reducing the prediction error compared with a single model, and ensuring the precursiveness and accuracy of the migration decision; Second, the key lineage task weight ratio constraint introduced in the migration trigger condition screens high-priority tasks through the lineage influence coefficient, solving the problem of the decline in system stability caused by the traditional overload migration strategy blindly migrating key tasks. For example, in a task link A-B-C, if the lineage influence coefficient I A = 125.72 exceeds the threshold I th = 120, then its weight ratio reaches 79.4% (θ = 0.75). After triggering the migration, the low-lineage influence task T3 is preferentially migrated rather than task A, ensuring the resource continuity of key tasks; Further, through the non-linear coupling of the bandwidth sensitivity coefficient and the total cluster bandwidth in the migration utility function, the load reduction amount and the bandwidth release effect are dynamically balanced when calculating the migration benefit. For example, when B t = 500 MB, the available bandwidth increment ΔB = 100 MB only contributes a utility value of ln(1.2) ≈ 0.182, avoiding the intensification of resource competition for other tasks caused by excessive bandwidth resource tilt; In addition, the benefit amplification coefficient models the sensitivity of the task deadline through the exponential gain factor. When the task slack time is much greater than the execution time, the migration utility can be effectively amplified, ensuring that high-priority tasks obtain the priority migration right, while the benefit amplification coefficient of tasks near the deadline will only be slightly amplified, effectively suppressing the risk of blind migration.

[0044] S600: Form a dynamic feedback control mechanism by periodically updating the lineage graph, adjusting the calculation parameters of the influence coefficient, optimizing the parameters of the time series prediction model, and evaluating the migration strategy.

[0045] Trigger the graph update based on a preset period or a graph update threshold. For the dynamic adjustment of the edge weight, based on the dynamic weight calculated in step S100, combined with the latest collected execution logs and flow metadata, optimize the adaptive decay coefficient through the online gradient descent algorithm. Based on the forward / backward propagation model in step S200, recalculate the lineage influence coefficient and update the set of key lineage tasks. Exemplarily, in a task link A-B-C, if the data volume of the downstream task C of task B suddenly increases, then dynamically adjust the edge weight W BC , and recalculate the lineage influence coefficient I B of task B. If I B exceeds the threshold I th , then include it in the set K of key lineage tasks.

[0046] Perform online optimization on the gain coefficient and attenuation factor to dynamically adjust the backpropagation gain coefficient and exponential attenuation factor through the online gradient descent algorithm, minimizing the prediction error of the historical blood relationship influence coefficient; perform adaptive update on the task level coefficient, and dynamically adjust the task level coefficient based on the actual execution performance of the task, such as the task elasticity coefficient or resource occupancy ratio.

[0047] Update the dynamic mixing coefficient of the hybrid model, and dynamically adjust the mixing coefficient through the online gradient descent algorithm based on the load prediction to minimize the prediction error.

[0048] Calibrate the parameters of the migration utility function, dynamically adjust the parameters of the migration utility function based on the task delay and resource recovery efficiency after migration; at the same time, adjust the threshold for migration triggering based on the change in the proportion of the weights of the key blood relationship tasks after migration. For example, if the resource guarantee for the key tasks is insufficient after migration, lower the threshold to expand the coverage of the key tasks.

[0049] In summary, the intelligent resource scheduling method based on the dynamic data lineage graph disclosed in this embodiment realizes the collaborative improvement of resource scheduling efficiency and fairness through the full-link closed-loop optimization of dynamic data lineage graph construction, lineage influence coefficient propagation model, comprehensive priority scoring, resource allocation strategy, and migration decision-making. First, a dynamic directed weighted lineage graph is constructed based on task execution logs and stream metadata. By real-time collecting multi-dimensional feature vectors (including data volume, transmission delay, and task dependency strength), combined with the dynamic weight allocation model, the time-delay sensitivity, scale effect, and task correlation of data dependencies are quantified, breaking through the limitations of traditional static lineage analysis and enabling the weight calculation to respond in real time to sudden changes in task dependencies. Second, the lineage influence coefficient is calculated through the weighted propagation model of the forward dependency path and the reverse dependency path, comprehensively considering the influence of tasks on downstream nodes and the constraint effect of upstream nodes, and introducing tree search to optimize the propagation path, effectively improving the accuracy of the propagation path compared with the traditional PageRank algorithm. Further, the comprehensive priority scoring module realizes the adaptive adjustment of task priorities through the dynamic weight allocation mechanism and the non-linear utility function, solving the problem of resource skew in the fixed weight scheme during sudden load changes. The resource allocation module divides the thresholds into three levels of high / medium / low priority and has a low-priority task resource recycling mechanism, which ensures the resource upper limit of high-priority tasks while avoiding low-priority tasks from being completely deprived due to resource competition. The migration decision-making module adopts a hybrid load prediction method that integrates deep learning models and statistical models, balances the non-linear fluctuation characteristics and linear trend modeling through the dynamic mixing coefficient, significantly reduces the prediction error compared with a single model, and combines a dual-condition trigger mechanism (predicted load overrun and key lineage task weight ratio overrun) to preferentially migrate tasks with low lineage influence and avoid interruptions of high-lineage tasks. Finally, through the four-dimensional closed-loop system formed by periodic updates of the lineage graph topology, online optimization of parameters, calibration of the time-series prediction model parameters, and evaluation of the migration strategy, the system can still maintain scheduling flexibility and critical task guarantee rate in the case of sudden load scenarios, breaking through the deficiencies of the prior art in the insufficient response to dynamic loads and rigid resource allocation, and finally achieving the dynamic balance of resource scheduling efficiency and fairness.

[0050] Reference Figure 2 , this embodiment further discloses an intelligent resource scheduling system based on a dynamic data lineage graph, including: A data acquisition module 21, configured to collect the execution logs and stream metadata of tasks in the computing platform in real time, and construct a dynamic directed weighted lineage graph based on the execution logs and stream metadata; including: capturing the execution logs and stream metadata of tasks in real time through a preset time window mechanism, extracting multi-dimensional feature vectors based on the execution logs and stream metadata, where the multi-dimensional feature vectors at least include data volume, data transmission delay, and task dependency strength; calculating the dynamic weights of the edges of the lineage graph through a weighted combination model based on the multi-dimensional feature vectors, and the calculation formula is: , where is the data volume in the data dependency relationship from the source node to the target node , is the transmission delay from the source node to the target node , is the dependency strength from the source node to the target node , , , and are adaptive attenuation coefficients; taking nodes as the vertices of the blood relationship graph and data dependencies as directed edges, a weighted blood relationship graph is formed through the dynamic weights.

[0051] The blood relationship influence coefficient calculation module 22 is used to calculate the blood relationship influence coefficient of each node in the dynamic directed weighted blood relationship graph through the weighted propagation models of the forward and reverse dependency paths; including: for all downstream nodes of the node , calculating the forward propagation influence value based on the sum of the products of the edge weights in the blood relationship graph and the preset task level coefficient ; for all upstream nodes of the node , calculating the reverse propagation influence value based on the sum of the power functions of the products of the edge weights in the blood relationship graph and the preset task level coefficient ; through the preset reverse propagation gain coefficient and the exponential decay factor , performing a weighted combination of the forward propagation influence value and the reverse propagation influence value to obtain the node blood relationship influence coefficient , and the calculation formula is: , where is the set of downstream nodes of the node , is the set of upstream nodes of the node , and are preset task level coefficients.

[0052] The comprehensive priority score calculation module 23 is used to obtain the task to be scheduled and calculate the comprehensive priority score based on the dependency weights of its dynamic blood relationship graph, the real-time load status of the target node, and the scheduling execution delay; including: obtaining the blood relationship influence coefficient of the task to be scheduled, the current load ratio of the target node, the scheduling execution delay[[ID=7,4]] and the task elasticity coefficient Parameters; dynamically update the blood relationship weights through a preset weight adjustment model , load weights and elastic weights ; fuse the parameters and weights through a multi-objective optimization function to calculate the comprehensive priority score, and the calculation formula is: , where is the delay attenuation factor.

[0053] Resource allocation module 24, which is used to perform priority sorting on the tasks to be scheduled based on the comprehensive priority score and allocate cluster resources according to the sorting result; including: sorting the comprehensive priority scores of the tasks to be scheduled in descending order to generate a priority queue , where is the highest priority task; according to the total amount of cluster resources and the task priority queue, dynamically set the resource allocation threshold, including: when the comprehensive priority score is greater than or equal to the preset score judgment threshold , set the high-priority resource threshold , when the comprehensive priority score is greater than or equal to the preset score judgment threshold and less than , set the medium-priority resource threshold When the comprehensive priority score is less than the low-priority resource threshold ; allocate resources to tasks in turn according to the order of the priority queue: when , set the task as a high-priority task and allocate resources to the high-priority task , where is the comprehensive priority score of node , is the total score of all tasks in the queue, when , set the task as a medium-priority task and allocate resources to the medium-priority task , when , allocate resources to the low-priority task , where is the sum of the resources already allocated to high-priority tasks and medium-priority tasks.

[0054] Load migration module 25, which is used to predict the load trend of the target node, and trigger a migration decision when it detects that the predicted load of the target node exceeds the threshold and the proportion of the weight of the critical blood relationship tasks it bears exceeds the preset threshold; including: predicting the future load of the target node based on a hybrid model of an integrated deep learning model and a statistical model , and the prediction function is: , where is the privacy-weighted load prediction value output by the deep learning model, is the dynamic mixing coefficient, is a constant term, and is the order of the statistical model, and are the autoregressive and moving average coefficients of the hybrid model, is the historical load value, is the historical residual term; Detect the predicted load of the target node and the proportion of the key lineage task weights. The calculation formula for the proportion of the key lineage task weights is: where is the set of key lineage tasks, is the target node is the set of tasks currently carried by the target node, is the target node is the total weight of the key lineage tasks on the target node, is the target node is the total weight of all tasks on the target node; If the predicted load of the target node exceeds the preset threshold and the proportion of the key lineage task weights exceeds the preset threshold, then select the migration task and the target migration node based on the migration utility function and the revenue amplification mechanism.

[0055] Figure 3 is a schematic diagram of the physical structure of the electronic device provided by the embodiment of the present invention. As Figure 3 shown, the electronic device 50 includes: a processor 501 (processor), a memory 502 (memory), and a bus 503; Among them, the processor 501 and the memory 502 communicate with each other through the bus 503; The processor 501 is used to call the program instructions in the memory 502 to execute the methods provided by the above method embodiments.

[0056] This embodiment provides a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions cause the computer to execute the methods provided by the above method embodiments.

[0057] Those of ordinary skill in the art can understand that all or part of the steps to implement the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps including the above method embodiments; And the foregoing storage medium includes: various storage media such as ROM, RAM, magnetic disk, or optical disk that can store program codes.

[0058] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0059] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiments.

[0060] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. An intelligent resource scheduling method based on a dynamic data lineage graph, characterized in that, The method includes: Collecting the execution logs and flow metadata of tasks in the computing platform in real time, and constructing a dynamic directed weighted lineage graph based on the execution logs and flow metadata; Calculating the lineage influence coefficient of each node in the dynamic directed weighted lineage graph through the weighted propagation models of forward and reverse dependency paths; Obtaining the tasks to be scheduled and calculating the comprehensive priority score based on the dependency weights of their dynamic lineage graphs, the real-time load status of the target nodes, and the scheduling execution delay; Performing priority sorting on the tasks to be scheduled based on the comprehensive priority score, and allocating cluster resources according to the sorting results; Predicting the load trend of the target node, and triggering a migration decision when it is detected that the predicted load of the target node exceeds the threshold and the weight ratio of the critical lineage tasks it bears exceeds the preset threshold.

2. The intelligent resource scheduling method based on the dynamic data lineage graph according to claim 1, wherein The collecting the execution logs and flow metadata of tasks in the computing platform in real time, and constructing a dynamic directed weighted lineage graph based on the execution logs and flow metadata includes: Capturing the execution logs and flow metadata of tasks in real time through a preset time window mechanism, and extracting multi-dimensional feature vectors based on the execution logs and flow metadata. The multi-dimensional feature vectors at least include the data volume, data transmission delay, and task dependency intensity; Calculate the dynamic weight of the edges in the blood relationship graph through a weighted combination model based on the multi-dimensional feature vector, and the calculation formula is: , where is the amount of data in the data dependency relationship from the source node to the target node ; is the transmission delay from the source node to the target node ; is the dependency strength from the source node to the target node ; , , and are adaptive attenuation coefficients; Taking nodes as the vertices of the lineage graph and data dependency relationships as directed edges to form a weighted lineage graph through the dynamic weights.

3. The intelligent resource scheduling method based on the dynamic data lineage graph according to claim 1, wherein The calculating the lineage influence coefficient of each node in the dynamic directed weighted lineage graph through the weighted propagation models of forward and reverse dependency paths includes: For nodes all downstream nodes of , calculate the forward propagation influence value based on the sum of the products of the edge weights in the blood relationship graph and the preset task level coefficient ; For nodes all upstream nodes of , calculate the backpropagation influence value based on the sum of power functions of the product of the edge weights in the blood relationship graph and the preset task level coefficient ; Through a preset backpropagation gain coefficient and an exponential decay factor , the forward propagation influence value and the backpropagation influence value are weighted and combined to obtain a node blood relationship influence coefficient . The calculation formula is: , where is the set of downstream nodes of node , is the set of upstream nodes of node , and are preset task level coefficients 4. The intelligent resource scheduling method based on the dynamic data lineage graph according to claim 1, wherein, The obtaining the tasks to be scheduled and calculating the comprehensive priority score based on the dependency weights of their dynamic lineage graphs, the real-time load status of the target nodes, and the scheduling execution delay includes: Obtain the blood relationship influence coefficient of the task to be scheduled , the current load ratio of the target node , the scheduling execution delay and the task elasticity coefficient parameters; Dynamically update the consanguinity weight through a preset weight adjustment model , load weight and elasticity weight ; Fuse the parameters and weights through a multi-objective optimization function to calculate the comprehensive priority score. The calculation formula is as follows: , where is the delay attenuation factor.

5. The intelligent resource scheduling method based on the dynamic data lineage graph according to claim 4, wherein The performing priority sorting on the tasks to be scheduled based on the comprehensive priority score, and allocating cluster resources according to the sorting results includes: Sort the comprehensive priority scores of the tasks to be scheduled in descending order to generate a priority queue , where is the task with the highest priority; According to the total amount of cluster resources and the task priority queue, dynamically set the resource allocation thresholds, including: when the comprehensive priority score is greater than or equal to the preset score judgment threshold set the high-priority resource threshold when the comprehensive priority score is greater than or equal to the preset score judgment threshold and less than set the medium-priority resource threshold when the comprehensive priority score is less than set the low-priority resource threshold ; Allocate resources to tasks in sequence according to the order of the priority queue: When , set the task as a high-priority task and allocate resources to the high-priority task , where is the comprehensive priority score of node , is the total score of all tasks in the queue. When , set the task as a medium-priority task and allocate resources to the medium-priority task , when , allocate resources to the low-priority task , where is the sum of the resources already allocated to high-priority and medium-priority tasks.

6. The intelligent resource scheduling method based on the dynamic data lineage graph according to claim 1, characterized in that The predicting the load trend of the target node, and triggering a migration decision when it is detected that the predicted load of the target node exceeds the threshold and the weight ratio of the critical lineage tasks it bears exceeds the preset threshold includes: Hybrid Model Prediction of the Future Load of Target Nodes Based on an Integrated Deep Learning Model and a Statistical Model , and the prediction function is: , where is the privacy-weighted load prediction value output by the deep learning model, is the dynamic mixing coefficient, is the constant term, and are the orders of the statistical model, and are the autoregressive and moving average coefficients of the hybrid model, is the historical load value, is the historical residual term; Detect the predicted load of the target node and the proportion of the weights of critical lineage tasks. The calculation formula for the proportion of the weights of critical lineage tasks is as follows: , where is the set of critical lineage tasks, is the target node is the set of tasks currently carried by the target node, is the target node is the total weight of the critical lineage tasks on the target node, is the target node is the total weight of all tasks; If the predicted load of the target node exceeds the preset threshold and the weight ratio of the critical lineage tasks exceeds the preset threshold, then select the migration tasks and target migration nodes based on the migration utility function and the benefit amplification mechanism.

7. An intelligent resource scheduling system based on a dynamic data lineage graph, characterized in that, Includes: A data collection module, configured to collect the execution logs and flow metadata of tasks in the computing platform in real time, and construct a dynamic directed weighted lineage graph based on the execution logs and flow metadata; A lineage influence coefficient calculation module, configured to calculate the lineage influence coefficient of each node in the dynamic directed weighted lineage graph through the weighted propagation models of forward and reverse dependency paths; A comprehensive priority score calculation module, configured to obtain the tasks to be scheduled and calculate the comprehensive priority score based on the dependency weights of their dynamic lineage graphs, the real-time load status of the target nodes, and the scheduling execution delay; A resource allocation module, configured to perform priority sorting on the tasks to be scheduled based on the comprehensive priority score, and allocate cluster resources according to the sorting results; A load migration module, which is used to predict the load trend of a target node. When it detects that the predicted load of the target node exceeds a threshold and the weight ratio of the critical lineage tasks it bears exceeds a preset threshold, a migration decision is triggered.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1-6 above is implemented.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, the method described in any one of claims 1-6 above is implemented.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, the method described in any one of claims 1-6 above is implemented.

Images