Topology-aware visual scheduling method, system, device and readable medium

Abstract

This invention discloses a topology-aware visual scheduling method, system, device, and readable medium, belonging to the field of big data distributed scheduling and time-series intelligent prediction technology. The method includes the following steps: S1, initializing the transition tensor; S2, constructing a high-dimensional state vector; S3, collecting windows and smoothly completing time-series samples; S4, generating dynamic balance coefficients based on the DAG node scale, and simultaneously pruning low-probability terms of the tensor to alleviate dimensionality explosion; S5, dividing the prediction confidence into multiple risk levels; S6, automatically switching to a lightweight degradation mode to ensure basic monitoring; S7, generating a visual situation map of DAG risk transmission. The topology-aware visual scheduling scheme provided by this invention can predict operational risks such as cluster task congestion and resource overload in complex operation and maintenance scenarios such as DAG scheduling clusters, sudden fluctuations in business traffic, and hardware and software system anomalies, while reducing the computing resource overhead during model inference.

Description

Technical Field

[0001] This invention belongs to the field of big data distributed scheduling and time-series intelligent prediction technology, specifically involving a topology-aware visualization scheduling method, system, device, and readable medium. Background Technology

[0002] Distributed task scheduling platforms based on Directed Acyclic Graphs (DAGs) (Airflow, DolphinScheduler) have become core components of big data infrastructure. Existing scheduling systems generally incorporate time-series load prediction modules to anticipate node CPU and memory congestion and task backlog, enabling elastic resource scaling and early fault warnings.

[0003] However, existing technologies have the following shortcomings: existing time series prediction schemes mostly directly reuse the first-order Markov model for communication network traffic prediction. Communication data packets are short and upstream and downstream are independent, naturally satisfying the Markov assumption of no aftereffect. However, big data Spark, Hive, and SQL tasks have long execution cycles and strong DAG upstream and downstream dependencies. The current node congestion state is affected by the historical backlog of previous times and the execution progress of upstream parent nodes. The first-order Markov only relies on the state of the previous time moment for prediction, resulting in the loss of long-term dependencies and a rapid decline in prediction accuracy as the task chain lengthens.

[0004] Traditional solutions uniformly use a fixed 5-minute window to collect time-series indicators. During stable business periods, there is a lot of noise and large fluctuations. During peak hours, traffic surges occur, and the fixed window cannot quickly capture instantaneous risks, resulting in serious delays in early warning. At the same time, the existing entropy threshold is a fixed constant, which cannot adapt to the 7-day periodic fluctuations of the business and has poor adaptability.

[0005] Existing monitoring systems treat each scheduling node as an independent entity, ignoring the risk transmission effect of the DAG graph structure. Small delays upstream can be amplified step by step, causing a cascading failure of downstream batch tasks. The general PageRank algorithm is only used for static webpage weight calculation and has not been adapted for scenarios with multiple parent nodes, isolated nodes, and dynamic addition and deletion of tasks in the scheduling DAG. Furthermore, it has not embedded topological weights into the time series prediction model to correct the transition probability.

[0006] Higher-order Markov systems rely on multi-order historical states to improve prediction accuracy, but the state space expands exponentially with the order, resulting in problems such as dimensionality explosion, excessive memory consumption, and slow inference. Existing technologies lack clustering dimensionality reduction and sparse storage optimization schemes for scheduling discrete resource states, remaining only at the theoretical level and unable to be implemented.

[0007] The existing prediction and scheduling system is an open-loop structure, and the scheduling results cannot be used to optimize the prediction model parameters. When the prediction model's computing power overflows or nodes crash, the system directly loses its load prediction capability, retaining only basic monitoring without a lightweight degradation fallback solution. The damping coefficient, probability balance coefficient, and window threshold are all fixed constants and cannot be dynamically adjusted with the size of the DAG nodes and the concurrency of tasks, resulting in huge differences in prediction performance between small clusters and ultra-large clusters with thousands of nodes.

[0008] Therefore, there is an urgent need to provide a set of intelligent scheduling solutions that are fully adaptive and have self-iterative optimization capabilities to overcome the multiple coupling technical defects in existing big data DAG scheduling scenarios, such as the failure of the assumption of no aftereffect of the first-order Markov model, poor compatibility of the fixed sliding window, lack of topological risk transmission quantification, dimensional explosion of high-order models, static non-adaptive algorithm parameters, lack of self-optimization in predictive scheduling open loop, and lack of fallback degradation for model failures. Summary of the Invention

[0009] The purpose of this invention is to provide a topology-aware visualization scheduling method, system, device, and readable medium. This solution optimizes the full-link architecture of time-series prediction, topology calculation, and scheduling linkage, and can accurately predict cluster congestion risks under various business traffic, ultra-large DAG clusters, and system failure scenarios, thus expanding the applicability of big data scheduling platforms in complex batch task scenarios.

[0010] To achieve the above objectives, this invention provides a topology-aware visual scheduling method, comprising the following steps: S1. Analyze the DAG directed graph and use the improved PageRank algorithm adapted to the scheduling scenario to calculate the topological weight of each task node; cluster and compress the resource state space, and initialize the n-order Markov transition tensor using sparse storage; synchronously transfer the DAG graph data, topological weights, and n-order Markov transition tensor to S2 and S4, and at the same time transfer the optimal damping coefficient and node out-degree to S4. S2. Collect node resource indicators and adaptively determine the Markov optimal order based on the time series autocorrelation characteristics; discretize the resource indicators into multiple states, and integrate historical states and upstream topology weights to construct a high-dimensional state vector; pass the node resource indicators and time series autocorrelation coefficients to S3 and S4, and pass the high-dimensional state vector to S5. S3. Calculate the entropy of the time-series state sequence information, dynamically update the entropy threshold based on historical periodic entropy data, adaptively adjust the length of the time-series acquisition window according to the entropy threshold, and smoothly complete the window switching samples; synchronously transmit the Shannon entropy of the state sequence within the window, the dynamically updated chaos threshold, the adjusted window length, and the completed time-series samples to S4. S4. Iteratively update the Markov transition tensor based on window samples, generate balance coefficients adaptively with the DAG cluster size, and perform weighted correction on the original Markov transition probability by integrating topological weights; continuously update the clustering superstate and prune low probability terms of the tensor; pass the iteratively updated n-order transition tensor, the original Markov transition probability, and the corrected transition probability to S5, and pass the dynamic balance coefficients to S6. S5. Based on the corrected transfer probability, extrapolate the congestion probability of nodes in multiple time periods and calculate the prediction confidence. Combine the probability threshold and confidence to classify multiple risk levels. Transfer the extrapolated congestion conditional probability, prediction confidence, and classified risk levels to S6. The multi-time period prediction data is simultaneously transferred to S7. S6. Implement differentiated scheduling strategies based on different risk levels; collect real congestion data after scheduling and iteratively correct the topology weights in reverse; update the topology parameters after the weights converge; automatically switch to lightweight degradation prediction mode when the model calculation is abnormal, a node crashes, or memory overflows; send the corrected and converged topology weights back to S1 and S4, and transmit the fault degradation mode signal to S4 synchronously. S7 integrates DAG topology, prediction probability, and risk level to build a visualization model. Based on the node topology weight and risk level differentiation, it renders nodes and draws a risk transmission trend map to output to the front end. The topology weight, multi-time period prediction probability, and risk level data are synchronously stored in the storage layer.

[0011] Preferably, in step S1, the improved PageRank algorithm, clustering compression of the resource state space, and sparse storage method specifically include: Traverse the damping coefficient range and select the optimal damping coefficient; for multiple parent nodes, select the parent node. Maximum value and isolated nodes are fixed. K-means clustering is used to compress the state space, and COO sparse format is used for storage. The Markov transition tensor, representing the following: ; In the formula, For sparse storage The order of the transition tensor is used to reserve sparse storage bits to discard low-probability transition terms in order to solve the problem of dimension explosion. This is a vector of historical states; Predict the state vector for the next time step; The improved PageRank algorithm calculation formula is as follows: ; In the formula, For nodes PageRank weight values; This represents the total number of nodes in the DAG. The damping coefficient; For nodes The set of upstream neighbors; upstream node The degree of departure; upstream node PageRank weight values; Preferably, in step S2, the adaptive determination of the Markov optimal order, resource index discretization, and high-dimensional state vector based on time-series autocorrelation characteristics specifically includes: The optimal Markov order is determined as follows: ; In the formula, It is the optimal Markov order; for Time and The time series autocorrelation coefficient at time 1; The candidate timing delay order; When all correlation coefficients are <0.3, forced And limited ; Continuous resource indicators are discretized into four states: {idle, normal, congested, abnormal}. ; The formula for constructing a high-dimensional state vector is: ; In the formula, for The high-dimensional state vector at any given time; This represents the current discrete resource state. for The historical state at any given moment; This represents the maximum PR value of the upstream parent node.

[0012] Preferably, in step S3, the dynamic updating of the entropy threshold based on information entropy and historical periodic entropy data, the adaptive adjustment of the time-series acquisition window length, and the smooth completion of samples specifically include: The Shannon entropy is used to characterize the disorder of the window state. The entropy calculation formula is as follows: ; In the formula, The information entropy of the state sequence within the current window characterizes the degree of disorder in the system state; For state The probability of it appearing in the window; It is the base of the logarithm; Daily scrolling reads of entropy data from the past 7 days to calculate the entropy mean and standard deviation, dynamically generating high and low entropy thresholds. The calculation formula is as follows: : ; ; In the formula, High entropy threshold; Low entropy threshold; The average entropy over 7 days; The standard deviation of entropy over 7 days; Window length constraint The adjustment range for each instance shall not exceed ±120 seconds, and the window switching sample smoothing completion logic shall be executed. Preferably, in step S4, the adaptive generation of balance coefficients, the fusion of topological weights to correct the original transition probabilities, the updating of clustering superstates, and the pruning of low-probability tensor terms specifically include: Formula for calculating dynamic equilibrium coefficient: ; In the formula, This is the dynamic balance coefficient; This represents the current total number of nodes in the DAG; Topological weighted corrected transition probability formula: ; In the formula, This is the corrected final transition probability; The original Markov transition probability; Topological influence factor; Continuously update the clustering superstate and prune low-probability transition terms within the tensor.

[0013] Preferably, in step S5, the multi-level risk classification based on probability thresholds and confidence levels is specifically as follows: The formula for calculating risk level is: ; In the formula, The risk levels are categorized into high-risk with high confidence, medium-risk with low confidence, etc. For the congestion conditional probability deduced; To predict confidence levels; This refers to a node resource congestion event. For risk classification mapping function; Set two levels of congestion probability thresholds and a confidence threshold of 0.8, as follows: like and It was determined to be high-risk with a high confidence level; like and It was determined to be of medium risk with a low confidence level. in, The preset first probability threshold; This is the preset second probability threshold.

[0014] Preferably, in step S6, the differentiated scheduling strategy, the reverse iterative correction of topology weights, and the updating of topology parameters after weight convergence specifically include: In high-risk, high-confidence scenarios, downstream heavy tasks are blocked, CPU capacity is elastically expanded, and backpressure is sent upstream; in medium-risk, low-confidence scenarios, only the sampling frequency of indicators is increased and upstream tasks are downgraded. The convergence criterion for topological weights is: ; In the formula, for The mean of the topological weights after each iteration; incremental updates of the DAG topological parameters after weight convergence; The lightweight degradation prediction mode is as follows: the high-order tensor iteration calculation module is turned off, and the operation mode is switched to a first-order Markov model, a fixed-duration acquisition window of 300 seconds, and a static threshold alarm.

[0015] This invention also provides a topology-aware visual scheduling system, comprising: The topology parse tensor initialization module is used to parse the DAG directed graph. It uses an improved PageRank algorithm to calculate the topological weights of task nodes, clusters and compresses the resource state space, and initializes the nth-order Markov transition tensor with sparse storage. The topology parse tensor initialization module synchronously sends the DAG graph data, topological weights, and nth-order Markov transition tensor to the state construction module and the probability correction module, and at the same time sends the optimal damping coefficient and node out-degree to the probability correction module. The high-dimensional state vector construction module, connected to the topology analysis tensor initialization module, is used to collect node resource indicators, adaptively determine the Markov optimal order based on the time-series autocorrelation characteristics, discretize the resource indicators into multiple states, and fuse historical states and upstream topology weights to construct a high-dimensional state vector. The high-dimensional state vector construction module sends the node resource indicators and time-series autocorrelation coefficients to the dynamic window adjustment module and the probability correction module, and sends the high-dimensional state vectors to the risk inference and classification module. The entropy-driven dynamic window adjustment module, connected to the high-dimensional state vector construction module, is used to calculate the entropy of the time-series state sequence information, dynamically update the entropy threshold based on historical periodic entropy data, adaptively adjust the length of the time-series acquisition window, and perform smooth completion on the window switching samples. The entropy-driven dynamic window adjustment module sends the window Shannon entropy, chaos threshold, adjusted window length, and completed time-series samples to the probability correction module. The topology probability correction module, connected to the entropy-driven dynamic window adjustment module, is used to iteratively update the Markov transition tensor based on window time-series samples, adaptively generate balance coefficients and fuse topology weights to correct the original Markov transition probability, and continuously update the clustering superstate and prune low-probability terms of the tensor. The topology probability correction module sends the updated transition tensor, the original transition probability, and the corrected transition probability to the risk inference and classification module, and sends the dynamic balance coefficients to the scheduling closed-loop degradation module. The risk simulation and classification module, connected to the topology probability correction module, is used to infer the congestion probability of nodes in multiple time periods based on the corrected transfer probability and calculate the prediction confidence. It combines the probability threshold and confidence to classify multiple risk levels. The risk simulation and classification module sends the congestion condition probability, prediction confidence, and risk level to the scheduling closed-loop degradation module and pushes the multi-time period prediction data to the visualization module simultaneously. The scheduling closed-loop module, connected to the risk inference and classification module, is used to execute differentiated scheduling strategies based on risk levels, collect real congestion data to iteratively correct topology weights, and update topology parameters after weight convergence; it switches to lightweight degradation prediction mode when model calculation is abnormal, node crashes, or memory overflow occurs; the scheduling closed-loop and fault degradation modules send the converged topology weights back to the topology parsing tensor initialization module and the topology probability correction module, and the fault degradation signal is synchronously sent to the topology probability correction module. The topology situation visualization module is connected to the scheduling closed-loop module. It is used to integrate DAG topology, prediction probability, and risk level to build a visualization model. It renders nodes differently according to topology weight and risk level and generates a risk transmission situation map to output to the front end. At the same time, it stores the topology weight, prediction probability, and risk level in the storage layer.

[0016] Therefore, by employing the aforementioned topology-aware visual scheduling method, system, device, and readable medium, the present invention offers the following advantages compared to existing technologies: (1) The present invention adopts a topology weight fusion adaptive high-order Markov chain and entropy-driven dynamic window coupling architecture to overcome the defects of a single model that cannot take into account the long-term dependence of tasks and the lag in the early warning of sudden traffic, so as to realize the early prediction of global congestion and the collaborative prediction effect that cannot be achieved by a single solution through multi-module linkage. (2) This invention adopts a self-learning entropy threshold, adaptive model order, dynamic balance coefficient, tensor lightweighting, and DAG incremental weight calculation integrated mechanism to overcome the problems of fixed parameters, high-order model computing power explosion, and high large graph computing cost, adapt to various scheduling business scenarios, and balance the needs of stable noise reduction and sudden risk capture. (3) This invention adopts a topology weight global risk quantification, scheduling feedback closed-loop iteration, and model failure degradation fallback mechanism to overcome the defects of traditional systems such as isolated monitoring, lack of self-optimization, and failure upon anomaly. It realizes full-link risk transmission control, continuously optimizes prediction capabilities, and ensures the stable operation of basic monitoring under system anomalies.

[0017] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating an embodiment of a topology-aware scheduling method according to the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Unless otherwise defined, the technical or scientific terms used in the present invention should have the ordinary meaning understood by those skilled in the art.

[0020] Example 1 like Figure 1 As shown, this embodiment provides a topology-aware visual scheduling method, system, device, and readable medium. It should be understood that the specific parameters, models, and protocols mentioned in this embodiment are merely examples to help those skilled in the art understand the present invention, and are not intended to limit the present invention.

[0021] The present invention provides a topology-aware scheduling method, comprising the following steps: S1. The DAG directed graph data obtained by parsing, the improved PageRank topological weights, and the initialized n-order Markov transition tensor are passed to steps S2 and S4; at the same time, the calculated optimal damping coefficient and node out-degree data are synchronously passed to step S4.

[0022] In this step, DAG topology parsing and tensor initialization are completed. Unlike the traditional approach that only focuses on task-dependent logic, this step simultaneously introduces an improved PageRank algorithm to pre-compute topology weights, providing a foundation for topology probability correction in subsequent high-order Markov predictions.

[0023] Analysis of the Directed Acyclic Graph (DAG) of Business Scheduling Task Flow , For a set of task nodes, For upstream and downstream dependent edges; traverse By comparing the topological weights with the MSE error, determine To find the optimal damping coefficient; utilize multiple parent nodes and take all parent nodes. The maximum value is used as the upstream influence factor, and there are no isolated upstream and downstream nodes. A special node weight adaptation strategy is used to calculate the node topology weight; K-means clustering is used to pre-compress the state space, and the n-order Markov transition tensor is initialized with COO sparse storage format.

[0024] The basic formula for constructing a node's PageRank is expressed as: ; In the formula, For nodes The PageRank weight value indicates the importance of a node in the scheduling network; This represents the total number of nodes in the DAG. The damping coefficient; For nodes The set of upstream neighbors; upstream node The degree of departure; upstream node PageRank weight value.

[0025] initialization The Markov transition tensor is expressed as: ; In the formula, For sparse storage The order of the transition tensor is used to reserve sparse storage bits to discard low-probability transition terms in order to solve the problem of dimension explosion. This is a vector of historical states; Predict the state vector for the next time step.

[0026] The system completes the reading of DAG graph data and pre-calculation of weights at a sampling frequency of 1Hz, and simultaneously completes state space compression and tensor storage format initialization.

[0027] The topological weight data and transition tensor structure output in this step provide the basic input for subsequent full-link index collection, high-dimensional state vector construction, and high-order Markov prediction inference. S2. Full-link indicator collection and adaptive n-order high-dimensional state vector construction: The real-time collected node CPU, memory, task backlog, and calculated autocorrelation coefficient are passed to steps S3 and S4; at the same time, the constructed high-dimensional state vector is synchronously passed to step S5.

[0028] In this step, the collection of full-link resource indicators and the construction of state vectors are completed. Unlike the traditional method of collecting resource snapshots at a single point in time, this step is based on time-series autocorrelation analysis to adaptively fuse multiple levels of historical states, providing a foundation for accurately capturing the trend of congestion evolution in the future.

[0029] The end-to-end vector data collector collects continuous resource metrics such as CPU, memory, and task backlog at each node in real time; it also calculates the time-series autocorrelation coefficient. The order is taken from the maximum time delay with a correlation coefficient ≥ 0.3. .

[0030] If all correlation coefficients are <0.3, then mandatory and restrictions Discretize continuous resource indicators into four states: {idle, normal, congested, and abnormal}. By integrating multi-level historical states and topological weights, a high-dimensional state vector is generated.

[0031] Construct an adaptive high-dimensional state vector, represented as: ; In the formula, for The high-dimensional state vector at any given time represents the current resource state of a node and its historical evolution path; For time-series sampling time; This represents the current discrete resource state. for The historical state at any given moment; The maximum PR value of the upstream parent node represents the upstream influence factor.

[0032] The logic for calculating the time series autocorrelation coefficient is defined as follows: ; In the formula, It is the optimal Markov order; for Time and The time series autocorrelation coefficient at time 1; The candidate timing delay order; all Forced And limited ; The high-dimensional state vector and order information output in this step provide the basic input for subsequent variable-length window adjustment, high-order Markov tensor iterative update and multi-time period congestion probability inference. S3. Based on the adaptive variable-length window adjustment of the 7-day self-learning information entropy, the Shannon entropy of the state sequence within the window calculated in real time, the dynamically updated chaos threshold, and the adjusted window length are passed to step S4; at the same time, the supplemented time-series sample data are synchronously passed to step S4.

[0033] In this step, the window adaptive adjustment based on information entropy is completed. Unlike the traditional fixed-duration observation window method, this step introduces a 7-day self-learning mechanism and anti-jitter judgment strategy, which provides a foundation for balancing model sensitivity and computational cost in the future.

[0034] Real-time calculation of Shannon entropy for the state sequence within the window; daily early morning reading of the entropy time series of the past 7 days and calculation of the mean. Standard deviation Dynamically update the chaos threshold; set window hard constraints and anti-shaking judgment logic, and adjust the window length according to the entropy value change trend; execute smooth completion logic when switching windows.

[0035] The formula for calculating the Shannon entropy of a state sequence is as follows: ; In the formula, The information entropy of the state sequence within the current window characterizes the degree of disorder in the system state; For state The probability of it appearing in the window; The base of the logarithm is and its value is . .

[0036] The logic for dynamically updating the chaos threshold is defined as follows: ; ; In the formula, This is a high-entropy threshold used to determine whether to shorten the window; This is a low-entropy threshold used to determine whether to extend the window; The average entropy over 7 days; The standard deviation of entropy over 7 days.

[0037] The system adjusts the window length based on the continuous period determination logic. The adjustment range for a single adjustment shall not exceed ±120s.

[0038] The dynamic window length and smoothed time series samples output in this step provide the basic input for subsequent high-order Markov tensor iterative updates and dynamic balance coefficient calculations. S4, Higher-order Markov tensor iterative update, dynamic Coefficients and topological probability corrections are iteratively updated. The order transition tensor, the calculated original Markov transition probability, and the final corrected transition probability are passed to step S5; simultaneously, the balance coefficient is dynamically calculated. The data is then transmitted synchronously to step S6.

[0039] This step involves parameter iteration and probability correction of the high-order Markov model. Unlike the traditional approach that relies solely on historical statistical frequencies, this step incorporates dynamic... The coefficients and topology weights are weighted and corrected to provide a foundation for improving the accuracy of congestion prediction and topology perception capabilities.

[0040] Iterative updates using the current window time series samples The transition tensor is normalized to obtain the original Markov transition probability; the dynamic balance coefficient is adaptively calculated by combining the current total number of nodes in the DAG with the real-time concurrency. The final transition probability is corrected by integrating topological weights; K-means clustering is continuously used to update the state and superstate, and low-probability terms of sparse tensors are pruned.

[0041] The adaptive calculation formula for the dynamic equilibrium coefficient is constructed as follows: ; In the formula, The dynamic equilibrium coefficient represents the fusion ratio of Markov predictions and topological weights. This represents the current total number of nodes in the DAG.

[0042] The topology weighting correction logic is defined as follows: ; In the formula, This is the corrected final transition probability; The original Markov transition probability; This is the topological influence factor.

[0043] The system monitors concurrency in real time, and when the real-time concurrency exceeds 1000, it will... Additional decay of 0.2, and continuous control of memory usage.

[0044] The corrected transition probability and updated tensor data output in this step provide the basic input for subsequent multi-period congestion probability extrapolation and risk confidence level determination. S5. Multi-period congestion probability extrapolation and risk confidence level classification determination: The extrapolated future multi-period node congestion conditional probability, output prediction confidence level and classified risk level data are transmitted to step S6; at the same time, the multi-period prediction data are transmitted to step S7.

[0045] In this step, congestion probability projection and risk classification are completed for multiple time periods. Unlike the traditional approach that relies solely on simple threshold alarms, this step combines confidence assessment and multi-time period projection results to provide a foundation for the selection of subsequent hierarchical adaptive scheduling strategies.

[0046] Input the current high-dimensional state vector into the high-order Markov prediction and inference unit; use the corrected... The transition tensor is used to extrapolate the probability of congestion and abnormal conditions at nodes in the future over multiple time periods; the confidence level of the prediction results is calculated; and the risk level is classified by combining the preset two-level probability thresholds and the confidence threshold of 0.8.

[0047] The risk assessment logic is constructed as follows: ; In the formula, The risk levels are categorized into high-risk with high confidence, medium-risk with low confidence, etc. For the congestion conditional probability deduced; To predict confidence levels; This refers to a node resource congestion event. This is a risk grading mapping function.

[0048] The classification criteria are defined as follows: like and It was determined to be high-risk with a high confidence level; like and It was determined to be of medium risk with a low confidence level. in, The preset first probability threshold; This is the preset second probability threshold.

[0049] The system performs inference calculations in real time and outputs detailed judgment results including probability values ​​and confidence levels.

[0050] The risk level and predicted probability data output in this step provide the basic input for subsequent execution of hierarchical adaptive scheduling, closed-loop weight iterative convergence, and visualized situation output. S6. Hierarchical adaptive scheduling, weighted convergence iteration, fault degradation and fault tolerance: execute scheduling instructions, collect real congestion results and determine the weighted data after convergence and transmit them to steps S1 and S4; at the same time, the fault degradation mode signal is transmitted to step S4.

[0051] In this step, hierarchical adaptive scheduling and closed-loop feedback are completed. Unlike the traditional passive alarm response method, this step executes differentiated scheduling strategies based on risk confidence and feeds back to correct model parameters, providing a foundation for subsequent system self-optimization and stable operation.

[0052] Tiered scheduling execution: For high-risk, high-confidence scenarios, downstream high-consumption tasks are blocked, CPU capacity is elastically expanded, and backpressure is sent upstream; for medium-risk, low-confidence scenarios, only the sampling frequency of indicators is increased and upstream tasks are downgraded, without blocking tasks. Closed-loop weighted iterative convergence: The actual congestion results after scheduling execution are collected to correct the topology weights in reverse; the difference between the mean values ​​of adjacent weights is monitored to determine the convergence status. Fallback degradation logic: If tensor calculation timeout, node crash, or memory overflow is detected, the system automatically switches to a first-order Markov model, a fixed 300-second window, and a static resource threshold alarm mode.

[0053] The logic for determining weight convergence is constructed as follows: ; In the formula, for The mean of the topological weights after each iteration; 0.005 is the convergence threshold.

[0054] The fault degradation trigger condition is set as follows: like ,but ; In the formula, This refers to the system's operating status. This is a security downgrade mode.

[0055] The system monitors execution performance and system health in real time to ensure dynamic adjustment of scheduling strategies and service availability under extreme conditions.

[0056] The corrected topology weights and system operating status output in this step provide the basic input for subsequent DAG dynamic incremental updates, continuous model optimization, and global visualization monitoring.

[0057] S7, DAG topology association prediction situation visualization output The read topological weights, multi-time period prediction probabilities, and risk level data are transmitted to the front-end interactive layer for display; at the same time, the visualized situation map data is synchronously updated to the storage layer.

[0058] In this step, the visualization output of the DAG topology association prediction situation is completed. Unlike the traditional method of only displaying a real-time status list, this step integrates DAG dependencies, prediction probabilities and risk levels to draw a panoramic situation map, providing operation and maintenance personnel with an intuitive display of the global risk transmission path.

[0059] The front-end interaction layer reads topology weight data, multi-time period prediction probability data, and risk level data from the storage layer; parses the DAG directed graph structure and node dependencies; draws a visual situation map including DAG dependency connections, node topology weight rendering, congestion prediction time series curves, and risk level coloring; and provides an interaction interface for operations and maintenance personnel to view global risk transmission paths and node details.

[0060] Construct a visual situational data model, represented as follows: ; In the formula, For visualizing situational data objects; It is a set of nodes that includes position, weight, and state; A set of connections representing dependencies; For multi-period forecasting indicators; The node risk level.

[0061] The visualization rendering logic is defined as follows: ; ; In the formula, Render colors for nodes to map risk levels; Render the node size and map the topology weights.

[0062] The system refreshes the front-end interface in real time to ensure that maintenance personnel have access to the latest operational status of the scheduling system.

[0063] The visualization interface and interactive data output in this step provide intuitive basis for operations and maintenance personnel to monitor overall risks, evaluate scheduling effectiveness, and assist in decision-making.

[0064] Therefore, the present invention adopts the above-mentioned topology-aware visualization scheduling method, system, device and readable medium. This solution optimizes the full-link architecture of time series prediction, topology calculation and scheduling linkage, and can accurately predict the risk of cluster congestion under multi-type business traffic, ultra-large DAG clusters and system failure and abnormal scenarios, and expands the applicability of big data scheduling platform in complex batch task scenarios.

[0065] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0066] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A topology-aware visual scheduling method, characterized in that, Includes the following steps: S1. Analyze the DAG directed graph and use the improved PageRank algorithm adapted to the scheduling scenario to calculate the topological weight of each task node; The resource state space is clustered and compressed, and the nth-order Markov transition tensor is initialized using sparse storage. The DAG graph data, topological weights, and nth-order Markov transition tensor are synchronously transferred to S2 and S4, while the optimal damping coefficient and node out-degree are transferred to S4. S2. Collect node resource indicators and adaptively determine the Markov optimal order based on the time series autocorrelation characteristics; discretize the resource indicators into multiple states, and integrate historical states and upstream topology weights to construct a high-dimensional state vector; pass the node resource indicators and time series autocorrelation coefficients to S3 and S4, and pass the high-dimensional state vector to S5. S3. Calculate the entropy of the time-series state sequence information, dynamically update the entropy threshold based on historical periodic entropy data, adaptively adjust the length of the time-series acquisition window according to the entropy threshold, and smoothly complete the window switching samples; synchronously transmit the Shannon entropy of the state sequence within the window, the dynamically updated chaos threshold, the adjusted window length, and the completed time-series samples to S4. S4. Iteratively update the Markov transition tensor based on window samples, generate balance coefficients adaptively with the DAG cluster size, and perform weighted correction on the original Markov transition probability by integrating topological weights; continuously update the clustering superstate and prune low probability terms of the tensor; pass the iteratively updated n-order transition tensor, the original Markov transition probability, and the corrected transition probability to S5, and pass the dynamic balance coefficients to S6. S5. Based on the corrected transfer probability, extrapolate the congestion probability of nodes in multiple time periods, calculate the prediction confidence level, and combine the probability threshold and confidence level to classify multiple risk levels. The multi-period congestion conditional probability, prediction confidence, and risk level classification obtained from the simulation are transmitted to S6, and the multi-period prediction data are simultaneously transmitted to S7. S6. Implement differentiated scheduling strategies based on different risk levels; After collecting and scheduling real congestion data, the topology weights are iteratively corrected in reverse. The topology parameters are updated after the weights converge. When the model calculation is abnormal, a node crashes, or memory overflows, the lightweight degradation prediction mode is automatically switched. The corrected and converged topology weights are sent back to S1 and S4, and the fault degradation mode signal is synchronously transmitted to S4. S7. Integrate DAG topology, predicted probability, and risk level to build a visualization model. Render nodes based on node topology weights and risk level differences, and output a risk transmission trend diagram to the front end. Topological weights, multi-period prediction probabilities, and risk level data are synchronously stored in the storage layer.

2. The topology-aware visual scheduling method according to claim 1, characterized in that, In step S1, the improvements to the PageRank algorithm, clustering to compress the resource state space, and the sparse storage method specifically include: Traverse the damping coefficient range and select the optimal damping coefficient; for multiple parent nodes, select the parent node. Maximum value and isolated nodes are fixed. K-means clustering is used to compress the state space, and COO sparse format is used for storage. The Markov transition tensor, representing the following: ； In the formula, For sparse storage The order of the transition tensor is used to reserve sparse storage bits to discard low-probability transition terms in order to solve the problem of dimension explosion. This is a vector of historical states; Predict the state vector for the next time step; The improved PageRank algorithm calculation formula is as follows: ； In the formula, For nodes PageRank weight values; This represents the total number of nodes in the DAG. The damping coefficient; For nodes The set of upstream neighbors; upstream node The degree of departure; upstream node PageRank weight value.

3. The topology-aware visualization scheduling method according to claim 2, characterized in that, In step S2, the adaptive determination of the Markov optimal order, resource index discretization, and high-dimensional state vector based on time-series autocorrelation characteristics specifically includes: The optimal Markov order is determined as follows: ； In the formula, It is the optimal Markov order; for Time and The time series autocorrelation coefficient at time 1; The candidate timing delay order; When all correlation coefficients are <0.3, forced And limited ; Continuous resource indicators are discretized into four states: {idle, normal, congested, abnormal}. ; The formula for constructing a high-dimensional state vector is: ； In the formula, for The high-dimensional state vector at any given time; This represents the current discrete resource state. for The historical state at any given moment; This represents the maximum PR value of the upstream parent node.

4. The topology-aware visualization scheduling method according to claim 3, characterized in that, In step S3, the dynamic updating of entropy thresholds based on information entropy and historical periodic entropy data, adaptive adjustment of the time-series acquisition window length, and smooth completion of samples specifically include: The Shannon entropy is used to characterize the disorder of the window state. The entropy calculation formula is as follows: ； In the formula, The information entropy of the state sequence within the current window characterizes the degree of disorder in the system state; For state The probability of it appearing in the window; It is the base of the logarithm; Daily scrolling reads of entropy data from the past 7 days to calculate the entropy mean and standard deviation, dynamically generating high and low entropy thresholds. The calculation formula is as follows: : ； ； In the formula, High entropy threshold; Low entropy threshold; The average entropy over 7 days; The standard deviation of entropy over 7 days; Window length constraint The single adjustment range shall not exceed ±120s, and the window switching sample smoothing and completion logic shall be executed.

5. The topology-aware visualization scheduling method according to claim 4, characterized in that, In step S4, the adaptive generation of balance coefficients, the fusion of topological weights to correct the original transition probabilities, the updating of clustering superstates, and the pruning of low-probability tensor terms specifically include: Formula for calculating dynamic equilibrium coefficient: ； In the formula, This is the dynamic balance coefficient; This represents the current total number of nodes in the DAG; Topological weighted corrected transition probability formula: ； In the formula, This is the corrected final transition probability; The original Markov transition probability; Topological influence factor; Continuously update the clustering superstate and prune low-probability transition terms within the tensor.

6. The topology-aware visual scheduling method according to claim 5, characterized in that, In step S5, the multi-level risk classification, which combines probability thresholds and confidence levels, is specifically as follows: The formula for calculating risk level is: ； In the formula, The risk levels are categorized into high-risk with high confidence, medium-risk with low confidence, etc. For the congestion conditional probability deduced; To predict confidence levels; This refers to a node resource congestion event. For risk classification mapping function; Set two levels of congestion probability thresholds and a confidence threshold of 0.8, as follows: like and It was determined to be high-risk with a high confidence level; like and It was determined to be of medium risk with a low confidence level. in, The preset first probability threshold; This is the preset second probability threshold.

7. The topology-aware visualization scheduling method according to claim 6, characterized in that, In step S6, the differentiated scheduling strategy, the reverse iterative correction of topology weights, and the updating of topology parameters after weight convergence specifically include: In high-risk, high-confidence scenarios, downstream heavy tasks are blocked, CPU capacity is elastically expanded, and backpressure is sent upstream; in medium-risk, low-confidence scenarios, only the sampling frequency of indicators is increased and upstream tasks are downgraded. The convergence criterion for topological weights is: ； In the formula, for The mean of the topological weights after each iteration; incremental updates of the DAG topological parameters after weight convergence; The lightweight degradation prediction mode is as follows: the high-order tensor iteration calculation module is turned off, and the operation mode is switched to a first-order Markov model, a fixed-duration acquisition window of 300 seconds, and a static threshold alarm.

8. A topology-aware visual scheduling system, applied to the topology-aware visual scheduling method according to any one of claims 1 to 7, characterized in that, include: The topology parse tensor initialization module is used to parse the DAG directed graph. It uses an improved PageRank algorithm to calculate the topological weights of task nodes, and clusters and compresses the resource state space and initializes the nth-order Markov transition tensor with sparse storage. The topology analysis tensor initialization module synchronously sends the DAG graph data, topology weights, and n-order Markov transition tensor to the state construction module and the probability correction module, and at the same time sends the optimal damping coefficient and node out-degree to the probability correction module. The high-dimensional state vector construction module, connected to the topology analysis tensor initialization module, is used to collect node resource indicators, adaptively determine the Markov optimal order based on the time-series autocorrelation characteristics, discretize the resource indicators into multiple states, and fuse historical states and upstream topology weights to construct a high-dimensional state vector. The high-dimensional state vector construction module sends the node resource indicators and time-series autocorrelation coefficients to the dynamic window adjustment module and the probability correction module, and sends the high-dimensional state vectors to the risk inference and classification module. The entropy-driven dynamic window adjustment module, connected to the high-dimensional state vector construction module, is used to calculate the entropy of the time-series state sequence information, dynamically update the entropy threshold based on historical periodic entropy data, adaptively adjust the length of the time-series acquisition window, and perform smooth completion on the window switching samples. The entropy-driven dynamic window adjustment module sends the window Shannon entropy, chaos threshold, adjusted window length, and completed time-series samples to the probability correction module. The topology probability correction module, connected to the entropy-driven dynamic window adjustment module, is used to iteratively update the Markov transition tensor based on window time-series samples, adaptively generate balance coefficients and fuse topology weights to correct the original Markov transition probability, and continuously update the clustering superstate and prune low-probability terms of the tensor. The topology probability correction module sends the updated transition tensor, the original transition probability, and the corrected transition probability to the risk inference and classification module, and sends the dynamic balance coefficient to the scheduling closed-loop degradation module. The risk simulation and classification module is connected to the topology probability correction module. It is used to infer the node congestion probability in multiple time periods based on the corrected transfer probability and calculate the prediction confidence. It combines the probability threshold and confidence to classify multiple risk levels. The risk simulation and classification module sends the congestion condition probability, prediction confidence, and risk level to the scheduling closed-loop degradation module, and pushes multi-period prediction data to the visualization module simultaneously. The scheduling closed-loop module, connected to the risk inference and classification module, is used to execute differentiated scheduling strategies based on risk levels, collect real congestion data to iteratively correct topology weights, and update topology parameters after weight convergence; it switches to lightweight degradation prediction mode when model calculation is abnormal, node crashes, or memory overflow occurs; the scheduling closed-loop and fault degradation modules send the converged topology weights back to the topology parsing tensor initialization module and the topology probability correction module, and the fault degradation signal is synchronously sent to the topology probability correction module. The topology situation visualization module is connected to the scheduling closed-loop module. It is used to integrate DAG topology, prediction probability, and risk level to build a visualization model. It renders nodes differently according to topology weight and risk level and generates a risk transmission situation map to output to the front end. At the same time, it stores the topology weight, prediction probability, and risk level in the storage layer.

9. A computer device, characterized in that, include: A processor configured to be coupled to a memory, read and execute instructions and / or program code in the memory to perform the method as described in any one of claims 1-7.

10. A computer-readable medium, characterized in that, The computer-readable medium stores computer program code that, when executed on a computer, causes the computer to perform the method as described in any one of claims 1-7.

Images