Distributed computing power scheduling intelligent optimization method and system

Abstract

The invention relates to the technical field of distributed computing, and discloses a distributed computing power scheduling intelligent optimization method and system.The distributed computing power scheduling intelligent optimization method comprises the steps that a multi-level system dependency graph and a causal relationship model are constructed; a multi-dimensional anomaly detection and fault prediction technology is utilized to monitor the system state in real time and predict potential faults; when an anomaly is detected or a potential fault is predicted, performing key path analysis by combining a multi-level system dependency graph and a causal relationship model, identifying a fault propagation path and a key influence node, executing a fault isolation strategy, blocking fault propagation and implementing preventive resource scheduling; implementing intelligent repair and resource recombination; according to the method, early prediction, accurate positioning and efficient processing of faults in the distributed system are realized, allocation and scheduling of computing power resources are optimized, and the reliability, availability and performance of the system are improved.

Description

Technical Field

[0001] The present invention relates to the field of distributed computing technology, and more specifically, it relates to a method and system for intelligent optimization of distributed computing power scheduling. Background Art

[0002] With the continuous growth of the scale and complexity of distributed systems, fault diagnosis and resource scheduling in the system have become increasingly difficult. The following are the main problems in the prior art: Insufficient accuracy of fault location: Traditional fault diagnosis methods usually rely on single-dimensional threshold monitoring or simple statistical correlation analysis, making it difficult to accurately locate the root cause of faults in complex distributed systems, especially when there are complex dependencies among system components; Passive and lagging fault handling: Current fault handling strategies are mostly post-event reactive, starting to diagnose and repair only after the fault has occurred and affected the user experience, lacking effective prediction and prevention mechanisms; Separation of resource scheduling and fault handling: In the prior art, resource scheduling and fault handling are usually regarded as two independent issues. The resource scheduling system rarely considers the fault risks and health status in the system, resulting in low resource utilization efficiency in the face of faults; Severe dependence on manual intervention: The diagnosis and repair of complex faults still highly rely on the experience and intuition of operation and maintenance personnel, making it difficult to achieve intelligent and automated processing, resulting in an extended fault repair time and reduced system availability; Lack of a systematic causal relationship model: The prior art mostly conducts analysis based on correlation rather than causality, being easily interfered by spurious correlations and making it difficult to accurately grasp the propagation path and influence scope of faults.

[0003] Therefore, there is a need for a distributed computing power scheduling optimization method that can organically combine fault management and resource scheduling and achieve proactive prevention and intelligent processing based on a causal relationship model. Summary of the Invention

[0004] The present invention provides a method and system for intelligent optimization of distributed computing power scheduling, which solves the technical problems of insufficient accuracy of fault location, passive and lagging fault handling, separation of resource scheduling and fault handling, and severe dependence on manual intervention in the related art.

[0005] The present invention provides a method for intelligent optimization of distributed computing power scheduling, including the following steps: Construct a multi-level system dependency graph and a causal relationship model. The multi-level system dependency graph includes a hardware layer, a virtualization layer, an application service layer, and a business function layer. The causal relationship model describes the dependency relationships and influence paths within and between each layer; Based on the multi-level system dependency graph and causal relationship model, using multi-dimensional anomaly detection and fault prediction technologies, the system status is monitored in real time and potential faults are predicted; When an anomaly is detected or a potential fault is predicted, key path analysis is performed in combination with the multi-level system dependency graph and causal relationship model to identify the fault propagation path and key impact nodes, and a fault isolation strategy is executed to block fault propagation and implement preventive resource scheduling; According to the fault isolation result and business continuity requirements, intelligent repair and resource reorganization are implemented.

[0006] In a preferred embodiment, the steps of constructing the multi-level system dependency graph and causal relationship model include: Collect component information and status data at all levels in the system; Analyze historical data through data mining techniques to identify the dependency relationships between components; Combining expert knowledge and machine learning algorithms, establish a causal relationship model between components; Regularly update the multi-level system dependency graph and causal relationship model to adapt to system changes.

[0007] In a preferred embodiment, the multi-dimensional anomaly detection and fault prediction steps include: Based on historical data and expert knowledge, establish a multi-dimensional normal behavior baseline for the system's key metrics; Monitor the system status metrics in real time, and use a variety of anomaly detection algorithms to identify abnormal patterns that deviate from normal behavior; Based on time series analysis and causal reasoning, predict the occurrence probability, impact range, and severity of potential faults; Generate anomaly detection and fault prediction reports to provide a basis for the next decision-making.

[0008] In a preferred embodiment, in the steps of blocking fault propagation and implementing preventive resource scheduling, the key path analysis and fault isolation include: Based on the multi-level dependency graph, analyze the propagation path of the anomaly or potential fault; Calculate the risk assessment results of each propagation path, and identify the key nodes and links that need to be processed first; Formulate and execute targeted isolation measures to block fault propagation; Implement a hierarchical isolation strategy for services with high availability requirements to ensure the continuity of core services.

[0009] In a preferred embodiment, the preventive resource scheduling steps include: Based on the fault risk assessment results, adjust the resource allocation strategy; Perform a health score on tasks at high-risk nodes, and calculate the migration priority score based on the scoring results and priorities; Create shadow instances for critical tasks to achieve seamless switching; Implement multi-level resource reservation and priority management to ensure the resource requirements of high-priority tasks.

[0010] In a preferred embodiment, the intelligent repair and resource reorganization step includes: Automatically select an appropriate repair strategy according to the fault type and system status; Execute an automatic repair process for faults that can be automatically repaired; For complex faults that require manual intervention, generate a detailed fault report and repair suggestions; Reallocate and optimize resource configuration according to the system status after repair and business requirements.

[0011] In a preferred embodiment, a distributed computing power scheduling intelligent optimization method further includes continuous learning and optimization steps: Collect and analyze data and results during the fault handling process; Update the causal relationship model and anomaly detection baseline; Optimize the fault prediction algorithm and resource scheduling strategy; Generate an optimization report for long-term system improvement.

[0012] In a preferred embodiment, a distributed computing power scheduling intelligent optimization system for executing a distributed computing power scheduling intelligent optimization method includes: A system perception and modeling module, which is used to collect component information and status data at all levels of the system, construct a multi-level system dependency graph and a causal relationship model, and monitor the system status in real time, identify anomalies and predict potential faults; A fault analysis and isolation module, which is used to analyze the fault propagation path, identify key nodes, formulate and execute targeted isolation strategies to block fault propagation; A resource scheduling and repair module, which is used to perform preventive resource scheduling based on the risk assessment results and implement intelligent repair strategies according to the fault type and system status; A continuous learning and optimization module, which is used to collect and analyze fault handling data, update the causal relationship model and anomaly detection baseline, and continuously optimize the system model and scheduling strategy.

[0013] In a preferred embodiment, a distributed computing power scheduling intelligent optimization system further includes a decision support module, which is used for: Comprehensively analyze anomaly detection, fault prediction and resource status information; Evaluate the costs and benefits of different processing strategies; Automatically or semi-automatically make processing decisions based on preset strategies and system status; Record the decision-making process and results for subsequent learning and optimization.

[0014] In a preferred embodiment, the distributed computing power scheduling intelligent optimization system is deployed using a distributed architecture, including: Central control node, responsible for global decision-making and coordination; Multiple edge nodes, distributed throughout the system, are responsible for local data collection and preprocessing; Data sharing mechanism ensures real-time synchronization of information between nodes; Fault-tolerant mechanism ensures that the system can still operate normally when some nodes fail.

[0015] The beneficial effects of the present invention are: Improved fault prediction capability: By building a multi-level system dependency graph and causal relationship model, combined with multi-dimensional anomaly detection technology, it is possible to identify potential faults in the system in advance, transforming fault handling from passive response to active prevention, and reducing the fault occurrence rate; Improved fault location accuracy: The root cause analysis method based on the causal relationship model can accurately trace the source and propagation path of the fault, shortening the fault location time from several hours in traditional methods to minutes, thus improving the location accuracy. Improved system availability: Preventive resource scheduling and fault isolation technology effectively block fault propagation and improve the availability of key services. Optimize resource utilization efficiency: Integrate fault risk perception into resource scheduling decisions to achieve dynamic optimization of resource allocation, improve resource utilization, and ensure system stability; Improved automation of operation and maintenance: Through intelligent repair and resource reorganization functions, manual intervention is greatly reduced, reducing operation and maintenance costs; Enhanced system resilience: Multi-level resource reservation and priority management mechanisms ensure the continuity of key services in the event of failures, reducing the time it takes for the system to recover from failures; Continuous optimization capability: By continuously learning from fault handling experience and system operation modes, the system's fault handling capability and resource scheduling strategy will continue to improve, forming an adaptive optimization closed loop. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of a distributed computing power scheduling intelligent optimization method of the present invention; Figure 2 It is a detailed flow chart of the present invention for constructing a multi-level system dependency graph and a causal relationship model; Figure 3 It is a detailed flow chart of the present invention for real-time monitoring of system status and prediction of potential failures; Figure 4 is a detailed flowchart for blocking fault propagation and implementing preventive resource scheduling in the present invention; Figure 5 is a detailed flowchart for implementing intelligent repair and resource reorganization in the present invention. Detailed implementation manners

[0017] The subject matter described herein will now be discussed with reference to example embodiments. It should be understood that discussing these embodiments is only to enable those skilled in the art to better understand and thus implement the subject matter described herein. Without departing from the scope of protection of the content of this specification, changes can be made to the functions and arrangements of the elements discussed. Each example can omit, substitute, or add various processes or components as needed. Additionally, the features described in some examples can also be combined in other examples.

[0018] In at least one embodiment of the present invention, a method for intelligent optimization of distributed computing power scheduling is disclosed, as Figures 1 to 5 shown, including the following steps: Step 1, construct a multi-level system dependency graph and a causality model. The multi-level system dependency graph includes a hardware layer, a virtualization layer, an application service layer, and a business function layer. The causality model describes the dependency relationships and influence paths within and between each layer; Specifically, it includes the following sub-steps: Step 1.1, collect system component dependency relationship data; Utilize an automatic discovery tool and a monitoring system API to collect the static configuration information and runtime dependency relationship data of each component in the distributed system. The collected data includes: Hardware component dependencies: the topological connections among physical servers, network devices, and storage devices; Infrastructure dependencies: the mapping relationships between virtual machines, containers, and physical hosts; Platform component dependencies: the call relationships among middleware such as databases, message queues, and caches; Application service dependencies: the dependencies among application layer components such as microservices and API call chains.

[0019] In some embodiments, by analyzing network traffic capture data, dynamic dependency relationships not recorded in the configuration management database can be automatically discovered, such as temporarily established data flow channels or implicit service calls. For example, in a microservices architecture, service discovery may not fully rely on a central registry, but rather identify the actual service call patterns through network traffic analysis, capturing a comprehensive dependency graph including occasional calls.

[0020] Step 1.2, construct a multi-level dependency graph; Construct a multi-level system dependency graph based on the collected dependency data : ; Among them, represents all components in the system; represents the dependency relationship between components; represents the dependency strength between components, and the larger the value, the stronger the dependency.

[0021] The dependency graph is organized through a hierarchical architecture, from the underlying hardware to the upper-layer applications, capturing the vertical dependency relationships between components at different levels and the horizontal dependency relationships between components at the same level, forming a complete dependency network.

[0022] Step 1.3, construction of the causal relationship model; Based on the Bayesian network and the theory of Structural Causal Model (SCM), upgrade the dependency graph to a causal relationship model. This causal model is constructed through the following steps: Apply the causal discovery algorithm to the historical monitoring data to initially identify the candidates for causal relationships between components; Combine the expert knowledge rule base to verify and adjust the causal relationships, and eliminate spurious correlations; Calculate the causal strength to quantify the magnitude of the causal impact between components; Establish a formal representation of the causal relationship: ; Among them, is the result variable, representing the state or performance metric of the system component being observed or predicted, is the set of all direct causal parent nodes of , representing all upstream components or factors that directly affect is the causal mechanism function, describing the mathematical mapping relationship of how the parent node affects the result variable through a specific mechanism, is the external interference factor, representing the random variable that cannot be observed in the model but affects , such as environmental noise, unknown interference, etc.

[0023] The specific implementation of the causal relationship model provided in this application is as follows: Hybrid causal discovery algorithm: This application adopts a hybrid causal discovery method, combining constraint-based algorithms (such as the PC algorithm) and score-based algorithms (such as the GES algorithm).

[0024] Among them, the PC algorithm first identifies possible causal relationships through conditional independence tests and constructs an undirected skeleton structure; then applies the orientation rules to determine the direction of the edges to form a partially directed graph.

[0025] The GES algorithm evaluates the likelihood of different graph structures through the Bayesian scoring function and searches for the optimal model in the search space.

[0026] Step 1.4, Causal discovery of multi-dimensional time series data; For the time series data generated by the system, apply specific causal discovery algorithms to identify causal relationships in the time dimension: Use the Granger Causality Test to analyze the causal relationships between time series data; Apply Conditional Transfer Entropy to calculate the direction and strength of information flow; Construct a Temporal Causal Graph to represent the changes in causal relationships under different time windows; Identify common causal chain patterns and summarize them into a fault propagation rule library.

[0027] The output of Step 1 is a complete multi-level system dependency graph and causal relationship model, which can accurately represent the dependencies and their causal strengths between components in a distributed system, providing a basis for subsequent root cause analysis and propagation prediction of faults.

[0028] Step 2, Based on the multi-level system dependency graph and causal relationship model, use multi-dimensional anomaly detection and fault prediction technologies to monitor the system status in real time and predict potential faults; Specifically, it includes the following sub-steps: Step 2.1, Deployment of distributed monitoring probes and data collection; Deploy lightweight monitoring probes at key nodes of the distributed system to continuously collect multi-dimensional runtime data: Hardware telemetry data: CPU temperature, fan speed, hard drive SMART information, etc.; System performance metrics: CPU utilization, memory usage, I / O latency, network throughput, etc.; System logs: operating system logs, application logs, error and warning messages; Performance counters: cache hit rate, number of lock competitions, garbage collection frequency, etc.; Network topology and traffic data: communication latency between nodes, packet loss rate, connection status changes.

[0029] In addition, the probes adopt an adaptive sampling strategy, dynamically adjusting the sampling frequency according to the system load and anomaly probability, minimizing the monitoring overhead while ensuring data quality.

[0030] In a resource-constrained edge computing environment, the probe can adopt a hierarchical activation mechanism, usually remaining in a low-power basic monitoring mode, and only activating the complete data collection function when abnormal signs are detected or a trigger signal from the central coordinator is received.

[0031] Step 2.2, multi-model anomaly detection system; Adopt an ensemble learning method, combine multiple anomaly detection algorithms, and capture different types of anomaly patterns: Apply a statistical model to detect anomalies in numerical metrics; Adopt a density-based model and a distance-based model to identify spatial anomaly points; Use a deep learning model to capture complex pattern anomalies in time series data and apply the IsolationForest algorithm to quickly identify global anomalies.

[0032] The outputs of each model are combined with confidence weighting to generate a comprehensive anomaly score, reducing false positives and false negatives.

[0033] Step 2.3, construction of a fault propagation dynamics model; This application provides a fault propagation model that draws on the theory of infectious disease transmission to quantify the diffusion process of faults in the system: Abstract system components as nodes with different states (healthy, latent, faulty); Define a state transition function: ; Among them, represents the state of component and at time , , respectively represent the states of component and component at time , is the set of components directly connected to component , is the parameter set; is the state transition function, describing how the state of the component evolves over time.

[0034] Establish a fault propagation rate matrix , matrix can be expressed as: ; Among them, , , respectively represent the propagation of a fault from component to component , component and component The rate; , , respectively represent the rate at which a fault propagates from component to component , component and component ; , , respectively represent the rate at which a fault propagates from component to component , component and component ; is the total number of components in the system. This rate value reflects the speed of fault propagation, and the larger the value, the faster the propagation.

[0035] The fault propagation dynamics model provided in this application is specifically implemented as follows: Implementation of the state transition function: The state transition function of this application adopts an improved SEIR (Susceptible-Exposed-Infected-Recovered) model framework, which is customized for the characteristics of the computing system.

[0036] The complete state transition function is: ; where, represents the state value of component i at time t + 1, represents the current state value of component i at time t, is the global propagation parameter, a coefficient that controls the overall fault propagation speed, represents the set of all components directly connected to component i, is the specific propagation rate from node j to node i, representing the specific rate at which a fault propagates from j to i, is the fault state intensity of node j at time t, and the larger the value, the more serious the fault, is the fault self-generation rate, representing the probability coefficient of a component generating a fault by itself, is the load coefficient of node i, reflecting the current load level of the component. The higher the load, the greater the possibility of self-generation of a fault.

[0037] Step 2.4, Counterfactual reasoning and fault risk assessment; Apply counterfactual reasoning technology to evaluate the causal impact between system components through hypothetical interventions: Construct a counterfactual model based on structural equations: ; where, represents when the variable is intervened to the value At time the counterfactual value of variable is the counterfactual function used to calculate the result after intervention; is the intervention value for variable ; represents that under the background condition for variable the values of all other variables among its parents except ; represents the exogenous factors or random disturbances related to variable ; For each important component, calculate the probability of its impact on other components under failure conditions: ; where represents the probability function; represents the target component to be affected; represents intervening on component to make it in a failure state; represents the state of component when component is intervened to be in a failure state; represents the probability that component also fails when component fails.

[0038] Based on the impact probability and component importance, calculate the risk score for each component: ; where represents the risk score of component quantifying the potential impact degree of the failure of this component on the entire system; represents the set of all components in the system; represents any component in the system that may be affected by the failure of component ; represents the business importance of component ; represents the conditional probability that component also fails when component fails; represents the operation of artificially intervening on component to make it in a failure state.

[0039] In some embodiments, counterfactual reasoning can be extended to a multi-step reasoning process to evaluate the potential impact depth of the failure chain reaction.

[0040] For example, in addition to calculating direct impacts, secondary and tertiary chain reactions can also be evaluated, formalized as recursive counterfactual queries: ; where, represents the risk score of component X after considering n-step chain reactions; represents any component in the system that may be affected; represents the set of all components in the system; represents the business importance of component Y; represents the conditional probability that component Y fails after n-step chain reactions following the failure of component X; represents the operation of artificially intervening in component X to put it in a failed state; represents the number of steps of the chain reaction, used to distinguish direct impacts (n = 1) and deeper indirect impacts (n ≥ 2).

[0041] In practical applications, this multi-step reasoning can discover deep risk links that are difficult to identify by traditional methods, especially in highly complex microservice architectures.

[0042] The output of Step 2 includes: anomaly detection results, fault propagation models, node risk scores, and system risk heatmaps. These pieces of information together constitute a comprehensive assessment of the system health status, providing a basis for subsequent fault prevention and resource scheduling decisions.

[0043] Step 3, when an anomaly is detected or a potential fault is predicted, perform critical path analysis by combining the multi-level system dependency graph and the causal relationship model, identify the fault propagation path and critical impact nodes, and execute the fault isolation strategy to block fault propagation and implement preventive resource scheduling; Specifically, it includes the following sub-steps: Step 3.1, critical path analysis and fault isolation; Based on the multi-level dependency graph and risk assessment results, identify the critical nodes and links in the system and implement targeted isolation measures: Apply graph theory algorithms (such as the minimum cut set algorithm) to identify the critical path and "bottleneck points" of fault propagation; According to the connectivity and business importance of components, calculate the isolation value index of each node: ; where, represents the isolation value index of node used to quantify the cost-effectiveness of isolating this node; represents the i-th node or component in the system; represents isolating node The number of nodes that can prevent a fault from spreading, i.e., the total number of other nodes that can be protected after isolating this node; Indicates the number of important business nodes that may be affected by isolation, i.e., the scope of service interruption that may be caused by isolating this node; the higher this ratio, the greater the fault protection effect that can be obtained with less service impact by isolating this node.

[0044] Implement precise isolation for components with high IVI values to form a "firewall" to block the fault propagation path; Adopt a fusing mechanism to actively cut off non-critical dependencies communicating with it when an abnormality of a component is detected but before it completely fails, to prevent the spread of the abnormal state.

[0045] In some embodiments, critical path analysis can apply community detection algorithms to divide system components into multiple semi-autonomous functional communities, and then enhance isolation measures at the community boundaries. For example, in a large-scale microservices architecture, the modularity maximization algorithm (such as the Louvain method) can be used to identify groups of services with close cooperation and construct a multi-level fault isolation strategy based on the community structure.

[0046] In addition, in financial or medical systems with high availability requirements, the isolation mechanism can adopt a progressive strategy, starting from the "observation mode", gradually upgrading to the "flow limiting mode", and finally the full "isolation mode", with each stage based on the anomaly confidence and service impact assessment monitored in real time. This progressive isolation can balance fault protection and service continuity and reduce unnecessary service interruptions caused by misjudgment.

[0047] Step 3.2, Risk-aware resource allocation decision; This application provides a method for integrating risk assessment results into the resource scheduling decision-making process to avoid assigning tasks to high-risk nodes: Define the node health score: ; Among them, represents the health score of node used to quantify the reliability and stability of the node; represents the i-th node or component in the system; represents the fault risk score of node quantifying the likelihood of this node failing; the health score is inversely proportional to the risk score, and the higher the health score, the more stable and reliable the node; the value range of the health score is from 0 to 1, where 1 represents completely healthy (zero risk) and 0 represents extremely high risk.

[0048] According to the criticality of the task type and the characteristics of resource requirements, formulate a task node matching strategy. Maintain a "healthy node white list" for critical tasks to ensure that critical services are only assigned to nodes with a health score exceeding the threshold. ; Design a load balancing algorithm that considers failure risks to avoid overloading high-risk areas while balancing the system load. Implement a balance mechanism for task affinity and risk avoidance to optimize task assignment decisions. Step 3.3, Preventive resource migration. For tasks already running on high-risk nodes, proactively implement preventive migration and transfer them to healthy nodes: Calculate the migration priority score based on resource utilization and task priority: ; where represents the priority score for migrating task from node to node ; represents the i-th task in the system; represents the source node where the task is currently located; represents the target node to which the task will be migrated; represents task 's business priority, and the higher the value, the more important the task; represents the risk score of the source node , and the higher the value, the greater the risk of node failure; represents the health of the target node , and the higher the value, the more stable and reliable the node; represents the migration cost coefficient, with a value range of 0 to 1, and the higher the value, the greater the migration cost; represents the inverse indicator of the migration cost, and the higher the value, the more economically feasible the migration.

[0049] Implement a lightweight task state saving and recovery mechanism to support seamless migration of tasks between different nodes. Design an incremental migration strategy. For large tasks, first migrate the critical state and then gradually migrate the non-critical data to reduce the impact of migration on services. Establish a migration storm protection mechanism to limit the number of concurrent migration tasks and avoid system instability caused by a large number of simultaneous migrations.

[0050] In some embodiments, preventive resource migration can build a migration time prediction model using historical migration performance data to accurately evaluate the migration window in advance. For example, in the migration of stateful services, the system can use a regression model to predict the time required to complete the migration based on factors such as task type, data volume, and network bandwidth, and optimize the migration schedule accordingly to ensure the completion of critical task migration before the risk threshold is breached.

[0051] In addition, for critical tasks that cannot be migrated quickly (such as long-running data processing jobs), the system can adopt the "shadow instance" strategy; Start a replica instance of the task on a healthy node, run it in parallel with the original instance for a period of time without providing services externally, and then quickly switch the traffic at an appropriate time after the data and status synchronization are completed. This strategy is particularly applicable to scenarios with extremely high requirements for service continuity such as database master-slave switching.

[0052] Step 3.4, multi-level resource reservation and priority management; Build a multi-level resource reservation pool to allocate different levels of resource guarantees for tasks with different priorities; Implement strict resource reservation for the highest-priority tasks to ensure that there are sufficient resources to execute critical operations under any circumstances; Adopt a flexible reservation strategy for medium-priority tasks and dynamically adjust the proportion of reserved resources according to the system load and risk status; Implement a resource reservation recovery and reallocation mechanism. When the reserved resources are idle for more than the threshold, temporarily release them to low-priority tasks while retaining the preemption right; Build a resource scheduling priority map to quickly determine task priorities and make reasonable allocation decisions during resource competition.

[0053] The output of Step 3 is a set of proactive protection measures, including fault isolation strategies, risk-aware resource allocation decisions, preventive task migration plans, and multi-level resource reservation schemes. These measures together build a multi-layer defense line to prevent fault propagation and ensure the continuity of critical operations.

[0054] Step 4, implement intelligent repair and resource reorganization based on the fault isolation results and business continuity requirements; Specifically, it includes the following sub-steps: Step 4.1, root cause analysis of faults; Use a causal relationship model to locate the root cause of faults: Based on a multi-level system dependency graph, construct a causal graph (CausalGraph) of fault events; Apply a causal reasoning algorithm to the observed performance anomalies and alarm events to calculate the posterior probability of each potential root cause; Intervention analysis is performed using an optimized Pearl's do-calculus, and the strength of the causal relationship between potential root causes and observed symptoms is evaluated through virtual intervention (do-operator). Introduce temporal prior knowledge, consider event temporal constraints, and improve the accuracy of root cause inference.

[0055] Combine expert rules and data-driven methods to handle complex cascading failures and identify initial trigger events.

[0056] In some embodiments, root cause analysis can integrate uncertainty representation and reasoning capabilities to generate multiple hypothesized root causes and their probability distributions at the preliminary diagnosis stage, and continuously update the probabilities as new evidence is collected.

[0057] For example, the system can use a Bayesian belief network or a Markov logic network to represent the uncertainty of fault diagnosis and calculate conditional posterior probabilities using causal probability inference to support decision-making in the case of incomplete information.

[0058] Among them, represents the conditional probability that a specific root cause becomes the true fault cause under the condition of observing evidence ; represents the possible fault root cause variable; represents the th specific hypothesized fault root cause; represents the fault evidence variable collected by the system; represents the specific observed fault evidence value in the test environment.

[0059] In addition, the system can implement an adaptive diagnostic path to automatically select the most effective diagnostic strategy according to the initial symptom characteristics. For example, for network failures, the connection status and routing information are preferentially checked; for resource exhaustion failures, the resource usage trend and mutation points are preferentially analyzed. This context-aware diagnostic strategy selection can significantly reduce the average fault diagnosis time. In a large distributed system, the average root cause location time can be reduced from dozens of minutes by traditional methods to a few minutes.

[0060] Step 4.2, intelligent repair strategy generation; For the determined root cause, retrieve potential solutions from the repair strategy library; Use a method that combines case-based reasoning (CBR) and reinforcement learning to generate specific repair steps according to historical cases and the real-time system state; Establish a repair strategy evaluation model to calculate the comprehensive score for each candidate strategy; ; Among them, represents the comprehensive evaluation score of the repair strategy. The higher the score, the better the strategy; represents the i-th candidate repair strategy; represents the expected effect of the repair strategy. The higher the value, the better the repair effect; represents the implementation cost of the repair strategy, including consumption of computing resources, network bandwidth, etc.; represents the execution risk of the repair strategy. The higher the value, the greater the possible negative impact; represents the execution time of the repair strategy, that is, the time required from the start to the completion of the repair; and represent the weight coefficients of the expected effect, implementation cost, execution risk, and execution time, respectively. Considering the current system state and business requirements, dynamically adjust the evaluation weights, and preferentially select fast repair strategies in time-sensitive scenarios; Design a parallel repair task planning algorithm to identify repair steps that can be executed in parallel and shorten the total repair time; Step 4.3, Repair process monitoring and adjustment; Establish a monitoring system for key performance indicators (KPIs) of the repair process to evaluate the repair progress in real time; Define an expected repair effect model and set the expected system state improvement curve for each stage; Design an adaptive adjustment mechanism for the repair plan. When it is detected that the actual effect deviates from the expectation, automatically start plan optimization; Construct a prediction model for the scope of repair impact to evaluate the risk of secondary failures that may be caused by repair activities;

[0061] Implement a repair rollback trigger to be able to roll back in time when it is found that the repair operation has a negative impact. In some embodiments, the system can implement a semi-automatic repair framework for human-machine collaboration. For low-risk and standardized repair steps (such as restarting services, clearing caches), the system can execute them fully automatically; while for high-risk or non-standardized operations, the system generates detailed operation suggestions, which are executed after being reviewed by human experts.

[0062]

[0063]

[0064] ​​​​​​​The system can also learn from the manual operations of human experts and gradually improve its automatic repair ability by using operation logs and interactive feedback. This progressive automation strategy can gradually increase the degree of repair automation while ensuring safety.

[0065] In addition, the system repair process can implement multi-granularity rollback checkpoints, automatically creating system state snapshots before key repair steps to support fine-grained selective rollback. For example, before high-risk operations such as database schema changes, not only data backups are saved, but also the configuration states of all services dependent on the database are recorded to ensure that a full recovery to a safe state can be achieved when necessary.

[0066] Step 4.4, Resource reorganization and system optimization; Based on the fault characteristics and root cause analysis results, identify the vulnerable points and optimization opportunities in the system; Implement Resource Rebalancing to adjust the load distribution and avoid the risks of resource hotspots and single points of failure; Optimize the system dependency structure, reduce unnecessary cross-node dependencies, and improve the modularity and fault isolation ability of the system; According to historical fault data, implement resource redundancy or functional reconstruction for components that frequently experience faults; Establish a mechanism for elastic adjustment of the resource pool to dynamically adjust the allocation ratios of various resources according to the fault mode and business requirements.

[0067] In some embodiments, the system can perform simulation-based "hypothesis-validation" analysis. Before implementing major structural adjustments, first simulate and verify the effects of the optimization plan in the digital twin environment. For example, for a planned microservice architecture reorganization, the system can build a simulation model based on historical load and fault data to predict the performance and fault recovery ability after the reorganization. This simulation-driven optimization decision can significantly reduce the risk of architecture changes.

[0068] In addition, in a long-running system, resource reorganization can adopt a "progressive reconstruction" strategy, decomposing large-scale reorganizations into multiple independent and gradually implementable small changes. Observe the system performance and collect feedback after each change to adjust subsequent plans. This incremental optimization method is particularly suitable for critical business systems that cannot tolerate large-scale outages and can gradually enhance system resilience and fault resistance while ensuring business continuity.

[0069] The output of Step 4 is the fault root cause identification report, customized repair strategy, repair process monitoring data, and resource reorganization and optimization plan, which together achieve the rapid recovery of system faults and the continuous optimization of long-term health.

[0070] Application example of this embodiment: Taking a large financial service cloud platform as an example, this platform contains approximately 3,000 computing nodes and supports multiple key business systems such as transaction settlement, risk control analysis, and user services. The main challenges faced by the platform are: instability caused by a sharp increase in system load during peak trading periods, the risk of cascading failures caused by complex interdependencies between components, and long service interruptions under traditional fault handling methods.

[0071] Implementation process example: Example of constructing a multi-level system dependency graph and causal relationship model: First, the system collected dependency relationship data of more than 15,000 components including hardware devices, virtual machines, middleware, microservices, etc. Through API calls, log analysis, and network traffic monitoring, many dynamic dependency relationships not recorded in the configuration database were discovered.

[0072] When constructing the multi-level dependency graph, the system divided the components into four main levels: Hardware layer: 540 physical servers, 42 network devices, 38 storage devices; Infrastructure layer: 2,500 virtual machines, 780 container clusters; Platform layer: 120 database instances, 85 message queues, 60 cache clusters; Application layer: 650 microservices, 1,800 API endpoints.

[0073] The system applied a hybrid causal discovery algorithm to analyze the system operation data of the past 90 days and successfully identified approximately 28,000 pairs of causal relationships between components. In the expert knowledge fusion process, the system integrated 326 expert rules provided by the operation and maintenance team and eliminated approximately 4,200 pairs of component pairs that only showed correlation but no causal relationship.

[0074] Especially in the microservice cluster of the trading system, the system identified a key causal chain: the exhaustion of the connection pool of the payment gateway service will lead to an increase in the response delay of the transaction processing service, which in turn triggers the timeout retry of the user session management service, and ultimately leads to a sharp increase in database connections. This causal chain is often regarded as an independent problem in traditional correlation analysis, while this system clarified the propagation order and impact path of events through time-series causal analysis.

[0075] Example of multi-dimensional anomaly detection and fault prediction: The system deployed lightweight monitoring probes on all 3,000 computing nodes, and each probe collected 42 different performance metrics, including hardware metrics such as CPU, memory, I / O, network, etc., and software metrics such as process status and service response time.

[0076] In the anomaly detection process, the system integrated four different anomaly detection algorithms: Statistical model: used to identify anomalies in numerical metrics such as CPU usage and memory consumption; DBSCAN density clustering: used to discover anomaly points in the multi-dimensional metric space; Autoencoder: for time series data of performance metrics of critical services, capturing complex pattern changes; Isolation Forest: quickly screens global anomaly points and provides preliminary anomaly signals; For the transaction processing cluster, the system balances monitoring accuracy and system overhead by adopting an adaptive sampling strategy to increase the sampling frequency (once every 5 seconds) during peak trading periods and decrease the frequency (once every 30 seconds) during off-peak periods.

[0077] In a practical case, the system detected a slight but continuous increase in the query latency of a certain database cluster. Although it had not reached the traditional alarm threshold, the multi-model anomaly detection system identified that this pattern was highly similar to the precursors of historical failures. The system immediately initiated fault propagation prediction analysis and found that if no intervention was made, the problem would spread to the transaction processing services relying on this database within approximately 15 minutes.

[0078] Instances of fault propagation blocking and preventive resource scheduling: Based on the prediction results, the system first calculated the critical path of fault propagation and identified that the connection pool management component between the database cluster and the transaction processing service was the "bottleneck point" of fault propagation, with its isolation value index (IsolationValueIndex, IVI) reaching as high as 3.8. The system immediately implemented precise circuit breaker measures, routing non-critical query requests to standby read-only instances while keeping the core transaction functions unaffected.

[0079] Subsequently, the system initiated risk-aware resource scheduling, assigning nodes with a health score higher than 0.95 to all new transaction processing tasks. At the same time, for important tasks already running on nodes near the affected area, the system calculated the migration priority scores and identified that two key account reconciliation tasks needed to be migrated immediately.

[0080] The system adopted an incremental migration strategy, first migrating the memory state and critical data of the tasks (about 120MB) to enable the service to quickly resume on the new nodes, and then completing the migration of the remaining 4GB data in the background. During the entire process, the service interruption time did not exceed 800 milliseconds, far lower than the 2 seconds allowed by the Service Level Agreement (SLA).

[0081] At the same time, the system reserved an additional 20% of computing resources for high-priority transaction processing tasks and set preemption permissions to ensure that resources could be quickly acquired in case of sudden load increases.

[0082] Intelligent Repair and Resource Reorganization Example: After completing fault isolation and preventive migration, the system conducted an in-depth root cause analysis. Through cause-and-effect diagram analysis, the system identified that the root cause of the increased database query latency was the degradation of I / O performance in a specific disk partition of the storage subsystem. The system calculated the posterior probabilities of multiple possible root causes and determined that the disk performance degradation (probability 0.87) was the most likely cause, far higher than the connection pool configuration problem (probability 0.12) and network congestion (probability 0.08).

[0083] Based on the root cause analysis results, the system retrieved three potential solutions from the repair strategy library and calculated the comprehensive scores through the repair strategy evaluation model: Data Redistribution Solution: (High effect, low risk, but long time); Storage Hot Migration Solution: (Medium effect, low risk, medium time); Hardware Replacement Solution: (High effect, high risk, longest time).

[0084] The system selected the data redistribution solution, decomposed it into sub-tasks that could be executed in parallel, and continuously monitored the execution effect. During the repair process, the system detected that some table migrations caused changes in the query plan and timely adjusted the index strategy to avoid potential performance degradation.

[0085] After the repair was completed, the system optimized and reorganized the resource structure, adjusted the storage resource allocation strategy, increased the resource redundancy for I / O-intensive workloads, and optimized the dependency structure between the database connection pool and microservices to reduce cross-node calls.

[0086] Technical Effect Verification: This implementation method achieved remarkable technical effects in the actual application of this financial service cloud platform, mainly reflected in the following two aspects: Fault Prediction Accuracy and Lead Time: During the six-month operation period, the system successfully predicted 85% of potential system faults, and on average identified the fault symptoms 18.7 minutes in advance, providing an adequate time window for preventive measures. Table 1 shows the statistical results of the prediction effects for different types of faults: Table 1: Statistical Results of the Prediction Effects for Different Types of Faults; System Availability and Resource Efficiency: Through proactive prevention and precise isolation strategies, the system significantly improved service availability and optimized resource utilization efficiency. Table 2 shows the comparison data before and after applying this technical solution: Table 2: Comparison Data Before and After Applying This Technical Solution;

[0087] During a critical business peak period, the system successfully predicted and isolated a potential cascading fault chain, contained a serious accident that could have affected 85% of the transaction processing capacity within a local area, and only affected 2.3% of the non-core business, ensuring the continuity of core transaction operations and avoiding potential losses estimated to exceed one million US dollars.

[0088] From the above real application examples, it can be seen that the distributed computing power scheduling intelligent optimization method based on causal reasoning and multi-level dependency graphs in this embodiment can effectively predict and prevent system failures in the actual production environment, significantly improve system availability and optimize resource utilization efficiency, providing a powerful solution for fault management and resource scheduling in large-scale distributed computing environments.

[0089] The embodiments of the present invention have been described above. However, these embodiments are not limited to the above specific implementation manners. The above specific implementation manners are merely illustrative and not restrictive. Under the inspiration of this embodiment, those of ordinary skill in the art can also make more equivalent embodiments in various forms, all of which fall within the scope of protection of this embodiment.

Claims

1. A distributed computing power scheduling intelligent optimization method, characterized in that: The following steps are involved: Construct a multi-level system dependency graph and causal relationship model. The multi-level system dependency graph includes the hardware layer, virtualization layer, application service layer, and business function layer. The causal relationship model describes the dependencies and impact paths within and between each layer. Based on multi-level system dependency graphs and causal relationship models, multi-dimensional anomaly detection and fault prediction technology are used to monitor system status in real time and predict potential faults; When an anomaly is detected or a potential fault is predicted, a critical path analysis is performed by combining a multi-level system dependency graph and a causal relationship model to identify the fault propagation path and key impact nodes, and a fault isolation strategy is implemented to block the fault propagation and implement preventive resource scheduling; Implement intelligent repair and resource reorganization based on fault isolation results and business continuity requirements.

2. According to claim 1, a distributed computing power scheduling intelligent optimization method is characterized in that: The steps of constructing a multi-level system dependency graph and a causal relationship model include: Collect component information and status data at all levels in the system; Analyze historical data through data mining technology to identify the dependencies between components; Combine expert knowledge and machine learning algorithms to build a causal relationship model between components; Multi-level system dependency diagrams and causal models are regularly updated to adapt to system changes.

3. According to claim 1, a distributed computing power scheduling intelligent optimization method is characterized in that: The multi-dimensional anomaly detection and fault prediction steps include: Establish multi-dimensional normal behavior baselines for key system indicators based on historical data and expert knowledge; Monitor system status indicators in real time and use multiple anomaly detection algorithms to identify abnormal patterns that deviate from normal behavior; Predict the probability, scope and severity of potential failures based on time series analysis and causal reasoning; Generate anomaly detection and fault prediction reports to provide a basis for further decision-making.

4. According to claim 1, a distributed computing power scheduling intelligent optimization method is characterized in that: The critical path analysis and fault isolation in the steps of blocking fault propagation and implementing preventive resource scheduling include: Analyze the propagation path of abnormalities or potential faults based on multi-level dependency graphs; Calculate the risk assessment results of each propagation path and identify the key nodes and links that need to be treated first; Develop and implement targeted isolation measures to block the propagation of faults; Implement a hierarchical isolation strategy for businesses that require high availability to ensure the continuity of core businesses.

5. The distributed computing power scheduling intelligent optimization method according to claim 1 is characterized in that: The preventive resource scheduling step includes: Adjust resource allocation strategy based on failure risk assessment results; Perform health scores on tasks on high-risk nodes and calculate migration priority scores based on the score results and priorities; Create shadow instances for critical tasks to achieve seamless switching; Implement multi-level resource reservation and priority management to ensure resource requirements for high-priority tasks.

6. The distributed computing power scheduling intelligent optimization method according to claim 1 is characterized in that: The intelligent repair and resource reorganization steps include: Automatically select the appropriate repair strategy based on the fault type and system status; Execute automatic repair process for faults that can be automatically repaired; Generate detailed fault reports and repair suggestions for complex faults that require manual intervention; Reallocate and optimize resource configuration based on the repaired system status and business needs.

7. The distributed computing power scheduling intelligent optimization method according to claim 1 is characterized in that: It also includes continuous learning and optimization steps: Collect and analyze data and results during troubleshooting; Updated causality models and anomaly detection baselines; Optimize fault prediction algorithms and resource scheduling strategies; Generate optimization reports for long-term system improvement.

8. A distributed computing power scheduling intelligent optimization system, used to execute a distributed computing power scheduling intelligent optimization method according to any one of claims 1 to 7, characterized in that: include: System perception and modeling module, which is used to collect component information and status data at all levels of the system, build multi-level system dependency graphs and causal relationship models, and monitor system status in real time, identify anomalies and predict potential failures; Fault analysis and isolation module, used to analyze fault propagation paths and identify key nodes, formulate and implement targeted isolation strategies, and block fault propagation; Resource scheduling and repair module, which is used to perform preventive resource scheduling based on risk assessment results and implement intelligent repair strategies according to fault type and system status; The continuous learning and optimization module is used to collect and analyze fault processing data, update the causal relationship model and anomaly detection baseline, and continuously optimize the system model and scheduling strategy.

9. A distributed computing power scheduling intelligent optimization system according to claim 8, characterized in that: Also included are decision support modules for: Comprehensive analysis of anomaly detection, fault prediction, and resource status information; Evaluate the costs and benefits of different treatment strategies; Automatically or semi-automatically make processing decisions based on preset strategies and system status; Record the decision-making process and results for subsequent learning and optimization.

10. A distributed computing power scheduling intelligent optimization system according to claim 8, characterized in that: The distributed computing power scheduling intelligent optimization system is deployed using a distributed architecture, including: Central control node, responsible for global decision-making and coordination; Multiple edge nodes, distributed throughout the system, are responsible for local data collection and preprocessing; Data sharing mechanism ensures real-time synchronization of information between nodes; Fault-tolerant mechanism ensures that the system can still operate normally when some nodes fail.

Images