A server-network linkage fault self-recovery and traffic scheduling system for a private cloud database cluster

Abstract

The application discloses a kind of server-network linkage fault self-healing and traffic scheduling systems for private cloud database cluster, comprising: data acquisition layer, integrated topology layer, intelligent diagnosis layer, linkage self-healing and traffic scheduling layer and verification audit layer;Data acquisition layer realizes the standardization collection of server, network equipment, database cluster full-dimensional state data;Integrated topology layer is based on the standardized data of data acquisition layer, constructs server-network-database full-link dependence topology model, and realizes the dynamic update and visual display of topology;Intelligent diagnosis layer realizes the real-time detection of server, network, database full-dimensional exception, accurately determines fault type, root cause tracing and influence range identification, provides decision basis for linkage self-healing and traffic scheduling;Linkage self-healing and traffic scheduling layer realizes the atomization collaborative execution of database scheduling, network configuration update, traffic scheduling;Verification audit layer realizes the automatic validity verification after self-healing and traffic scheduling operation, forms operation and maintenance closed loop.The application can solve the technical defects of server-network-database separation management in existing private cloud database cluster operation and maintenance, realize three-layer collaborative control and intelligent traffic scheduling.

Description

Technical Field

[0001] This invention relates to the field of private cloud operation and maintenance technology, and in particular to a server-network linkage fault self-healing and traffic scheduling system for private cloud database clusters. Background Technology

[0002] In existing technologies, a private cloud database cluster refers to a distributed database service cluster deployed in an enterprise's private cloud environment, consisting of primary and backup database instances, server nodes, and network devices, which has data storage, read-write separation, and disaster recovery backup capabilities.

[0003] Atomized self-healing: refers to the execution of database scheduling operations and network configuration update operations as an inseparable whole, either all succeeding or all rolling back, ensuring operational consistency and business continuity.

[0004] Linked failures: These are complex failures triggered by failures at at least two levels in the server, network, and database, and their self-healing requires coordinated execution of multiple levels of operations.

[0005] In the operation and maintenance of database clusters deployed on private clouds, servers, networks, and databases generally adopt a separate management architecture: the server layer is independently managed by a hardware monitoring system, the network layer is configured and monitored by a network management platform, and the database layer is managed by database high-availability software for master-slave failover and instance management. There are no data interaction interfaces or collaborative execution mechanisms between the three. This presents the following problems:

[0006] (1) State awareness is isolated: The acquisition protocols and data formats of the monitoring systems at different levels are not uniform, and there is no global state synchronization mechanism. Problems such as server failure, network card abnormality, and network jitter cannot be synchronized to the database management system in real time, which makes it impossible for the database layer to predict risks in advance and to quickly perceive the root cause after a failure occurs.

[0007] (2) Lack of operational coordination: After the database master-slave switch and instance migration, the load balancing, routing policy, security group and ACL rules on the network side need to be manually updated. There is a time difference in operation, which can easily cause the business traffic forwarding to fail and cause business interruption. Moreover, manual operation has no atomic guarantee, which can easily lead to a half-complete state of "database switch completed but network configuration not updated", increasing the complexity of faults.

[0008] (3) Difficulty in root cause localization: Each level of monitoring system only outputs its own alarm information and has no fault correlation analysis mechanism. When a fault occurs, maintenance personnel need to investigate across systems. It is impossible to quickly distinguish whether the root cause is a problem with the server hardware / system, network link / equipment or the database itself. Troubleshooting time is usually more than 5 minutes, which is far from meeting the continuity requirements of enterprise-level business.

[0009] (4) Passive traffic scheduling: Existing traffic scheduling relies solely on static load balancing strategies at the network layer. It cannot dynamically adjust based on the operating status of the database instance (such as load, latency, primary / backup roles), the hardware status of the server node, and the quality of the network link. This can easily lead to traffic skew and reduce the overall performance of the database cluster.

[0010] (5) Low self-healing security: There is no unified verification and rollback mechanism. No automatic validity verification is performed after database or network configuration changes. If the change is incorrect, it will cause secondary failures. In addition, configuration change logs are scattered in various systems and there is no unified traceability system, which cannot meet the enterprise's compliance and internal control requirements.

[0011] (6) High dependence on manual operation: Database high availability depends entirely on manual intervention by operation and maintenance personnel, resulting in low recovery efficiency, high risk of misoperation, and a manual handling cost of approximately 500-2000 yuan for a single fault. Furthermore, it is impossible to achieve 24 / 7 unattended operation and maintenance.

[0012] In summary, existing technologies cannot achieve unified status awareness, collaborative operation execution, automatic fault root cause location, and dynamic traffic scheduling across the server-network-database layers. This makes it difficult to guarantee the high availability and automated operation and maintenance of private cloud database clusters, and an integrated, interconnected fault self-healing and traffic scheduling solution is urgently needed. Summary of the Invention

[0013] Purpose of the invention: This invention provides a server-network linkage fault self-healing and traffic scheduling system for private cloud database clusters, which can solve the technical defects of the existing private cloud database cluster operation and maintenance system with separate management of servers, networks and databases, and realize three-layer collaborative management and intelligent traffic scheduling.

[0014] Technical Solution: The present invention discloses a server-network linkage fault self-healing and traffic scheduling system for private cloud database clusters, comprising: a data acquisition layer, an integrated topology layer, an intelligent diagnosis layer, a linkage self-healing and traffic scheduling layer, and a verification and auditing layer. The data acquisition layer enables real-time, highly available, and standardized collection of full-dimensional status data of servers, network devices, and database clusters. The integrated topology layer constructs a server-network-database full-link dependency topology model based on the standardized data from the data acquisition layer, and realizes dynamic updates and visualization of the topology. The intelligent diagnosis layer enables real-time detection of anomalies in servers, networks, and databases, accurate fault type determination, root cause tracing, and impact scope identification, providing a decision-making basis for linkage self-healing and traffic scheduling. The linkage self-healing and traffic scheduling layer enables atomic collaborative execution of database scheduling, network configuration updates, and traffic scheduling. The verification and auditing layer enables automated validity verification after self-healing and traffic scheduling operations, full-process log recording, and automatic rollback of anomalies, forming a closed-loop operation and maintenance system.

[0015] Furthermore, the data acquisition layer includes a multi-protocol universal acquisition interface and a data cleaning and standardization module. The multi-protocol universal acquisition interface supports mainstream acquisition protocols such as SNMP, JDBC, Prometheus, SSH, and Telnet, and is compatible with servers, network devices, and databases from different brands. The data cleaning and standardization module performs deduplication, completion, and format conversion on the acquired raw data, and outputs it as structured standard data in formats such as JSON and XML, including but not limited to data identifiers, acquisition time, indicator values, and unique device IDs.

[0016] Furthermore, the integrated topology layer constructs a server-network-database end-to-end dependency topology model. The topology model includes three dimensions: the physical dimension, which is the physical connection relationship between server nodes and network devices; the network dimension, which is the logical connection relationship between network links, routing, load balancing, etc.; and the application dimension, which is the binding relationship between database instances and server nodes and network access policies. When devices are added / deleted, configurations are changed, or connection relationships are altered, the topology model is automatically updated with an update latency of ≤1 second. A topology visualization interface is provided: it supports graphical display of the end-to-end dependency relationship, and also provides an API interface for external systems to call.

[0017] Furthermore, the intelligent diagnostic layer includes a multi-dimensional anomaly detection module, a fault rule engine module, and a root cause localization inference model module. The multi-dimensional anomaly detection module comprehensively utilizes various algorithms to detect different types of indicators. For time-series indicators such as CPU utilization, link latency, and synchronization latency, a detection method combining a sliding window, the 3σ principle, and the EWMA exponentially weighted moving average algorithm is employed. This method first sets a configurable time window (default 60 seconds) containing 120 data sampling points. The algorithm calculates the weighted moving average of the data within the window, where the weight of historical data decays exponentially (smoothing coefficient α is set to 0.2). If the residual between the current actual value and the predicted value exceeds three times the standard deviation, and this state persists for more than three sampling points (i.e., 1.5 seconds), it is judged as an anomaly. For discrete status indicators such as server heartbeat, network card status, and database instance status, an algorithm combining rule matching and heartbeat timeout detection is used. This algorithm sets the heartbeat timeout threshold to 1 second and maintains a continuous timeout counter. After three consecutive heartbeat timeouts (cumulative 3 seconds), the state machine transitions the target state from "normal" to "abnormal." For cross-level correlated indicators, such as the correlation between server hard drive failure and database I / O anomalies, an algorithm combining Pearson correlation coefficient and causal inference is used. This algorithm calculates the Pearson correlation coefficient between the time series of two indicators within a 10-second time window. When the absolute value of the correlation coefficient is greater than a preset threshold (e.g., 0.85), further lag cross-correlation analysis is used to determine the causal direction. If it is confirmed that the change in one indicator (e.g., hard drive read / write latency) leads the change in another indicator (e.g., database I / O latency) in time, a strong causal relationship is determined, effectively filtering false alarms caused by secondary effects and controlling the false alarm rate below 0.1%.

[0018] Furthermore, the fault rule engine module constructs a custom rule library, containing four major categories of fault rules: server, network, database, and linkage. Each category of rules includes three core elements: indicator threshold, abnormal duration, and correlation conditions. A three-level priority determination mechanism is designed: based on the fault impact scope (single instance / single node / cluster level), business importance (core business / non-core business), and fault urgency (immediate self-healing / delayed self-healing / manual intervention), three levels of priority are set. Level 1 faults are cluster-level core business faults, triggering immediate self-healing; Level 2 faults are single-node non-core business faults, triggering delayed self-healing; and Level 3 faults are single-instance minor faults, only triggering alarms and not self-healing.

[0019] Furthermore, the root cause localization inference model module takes as input parameters: anomaly detection results, fault rule matching results, integrated topology model, and historical fault data. Based on the dependencies of the topology model, it starts from anomaly indicators, traces back to related devices / instances, and combines causal inference algorithms to filter secondary anomalies and locate the root cause of the fault. The core logic of this causal inference algorithm is as follows: The algorithm first takes all detected abnormal events as the starting point and performs a reverse depth-first traversal along the "dependent" direction in the integrated topology model to construct a complete "anomaly propagation tree". During the traversal, for any two nodes A (upstream) and B (downstream) with a direct dependency relationship, if node A has a clear anomaly (e.g., server hard drive failure), and node B's anomaly (e.g., database IO latency) occurs later than node A's anomaly, and there is a high-probability record (probability threshold > 0.85) of "A anomaly causes B anomaly" in the historical fault pattern library, then the algorithm determines that node B's anomaly is a "secondary anomaly" caused by node A's anomaly. The algorithm prunes all abnormal nodes marked as "secondary," identifying only the root node at the top of the propagation tree that has no upstream abnormal nodes as the final root cause of the failure. Output includes: root cause type, unique ID of the source device / instance, scope of impact, failure priority, and suggested self-healing strategy.

[0020] Furthermore, the integrated self-healing and traffic scheduling layer includes an atomic execution module, a database scheduling module, a network configuration update module, and a dynamic traffic scheduling module. The atomic execution module introduces distributed transaction locks and synchronous trigger signals to bind database scheduling, network configuration update, and traffic scheduling operations into an atomic transaction, ensuring "either all succeed or all roll back." The transaction timeout threshold is 5 seconds, triggering automatic rollback upon timeout. The database scheduling module supports two core operations: automatic master-slave failover and automatic instance migration, automatically selecting the operation type based on root cause analysis results. The migration strategy selects the optimal healthy node based on server node hardware load, remaining resources, and network topology. The network configuration update module executes synchronously with the database scheduling operations, automatically updating routing policies, load balancing configurations, security group rules, and ACL access control lists to ensure that network configuration matches the location / role of the database instance. During fault self-healing, the dynamic traffic scheduling module seamlessly switches business traffic to healthy database instances / server nodes. When load is skewed, a multi-dimensional weighted algorithm based on "database instance load + server hardware load + network link quality" is initiated. This algorithm comprehensively considers the load situation from three dimensions: the load of the database instance itself (e.g., number of connections, number of active transactions), the hardware load of the server node (e.g., CPU utilization, memory utilization), and the quality of the network link (e.g., latency, packet loss rate). Each dimension is assigned a configurable weight coefficient (default 1 / 3 for each) and normalized to between 0 and 1. The final service node weight is calculated using the formula Weight = w1*(1 - DB_load) + w2*(1 - CPU_usage) + w3*(1 - latency / max_latency). The system dynamically adjusts the traffic distribution ratio of each node in the load balancer backend pool based on this weight, achieving intelligent traffic routing. During primary / standby failover, the system switches 100% of write traffic to the newly elected primary database and distributes read traffic evenly to all healthy standby databases using a round-robin algorithm, achieving read / write separation optimization.

[0021] Furthermore, the verification and audit layer includes an automated verification module, an automatic rollback module, and an audit log module. The automated verification module simulates business requests to perform read / write tests, network link detection, and traffic monitoring. The verification timeout threshold is 3 seconds, and the number of verification attempts is 3 (rollback is triggered if failure occurs). The automatic rollback module adopts an atomic rollback mechanism to restore the database configuration, network configuration, and traffic scheduling policy to their pre-operation state, with a rollback execution delay of ≤2 seconds. After the rollback is completed, the fault is re-detected, and if it still exists, an alarm is triggered to notify the operations and maintenance personnel. The system stores data on a rolling basis for 90 days, supports retrieval by device ID, fault type, and time, and provides log export functionality to meet compliance and internal control requirements.

[0022] Beneficial Effects: Compared with existing technologies, this invention has the following significant advantages: It achieves unified perception and collaborative execution of the server-network-database three-layer states, solving the core defects of isolated states and disconnected operations in existing technologies; it binds database scheduling, network configuration updates, and traffic scheduling into atomic transactions, ensuring operational consistency, avoiding business interruption and secondary failures, and achieving seamless business recovery; the multi-algorithm fusion detection mechanism and root cause localization model enable real-time fault detection and accurate source tracing, significantly shortening fault troubleshooting time; the traffic scheduling strategy based on the three-layer states solves the traffic skew problem of existing static scheduling, improving the overall read / write performance and resource utilization of the database cluster; the automated verification and atomic rollback mechanism ensures the effectiveness of self-healing operations, avoids secondary failures caused by configuration changes, and improves system stability; the intelligent diagnosis and traffic scheduling layer adopts a plug-in design, supporting flexible expansion and dynamic updates of algorithms, rules, and scheduling strategies without modifying the existing architecture; it supports mainstream brand servers, network devices, and databases, without requiring large-scale modifications to existing private cloud architectures and database clusters, reducing implementation costs by more than 60%. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the system architecture of the present invention.

[0024] Figure 2 This is a schematic diagram of the method flow of the present invention.

[0025] Figure 3 This is a diagram of the server-network-database full-link dependency topology model of the present invention. Detailed Implementation

[0026] like Figure 1 As shown, a server-network linkage fault self-healing and traffic scheduling system for private cloud database clusters includes five core layers: data acquisition layer, integrated topology layer, intelligent diagnosis layer, linkage self-healing and traffic scheduling layer, and verification and auditing layer. Each layer achieves data interaction through standardized data interfaces, and the end-to-end processing latency is ≤2s.

[0027] The specific design of each level is as follows:

[0028] The core function of the data acquisition layer is to achieve real-time, highly available, and standardized collection of status data from servers, network devices, and database clusters across all dimensions, providing data support for subsequent topology construction, fault diagnosis, and traffic scheduling.

[0029] Data collection targets and specific metrics: Server layer: CPU utilization, memory utilization, hard disk status (health, IOPS), network card status, process status, and heartbeat signal of physical / virtual machines (collection frequency: 500ms); Network layer: Port bandwidth, link latency, packet loss rate, routing status, load balancing configuration, and security group / ACL rule status of switches / routers (collection frequency: 500ms); Database layer: Status of primary and standby instances, primary and standby synchronization latency, number of connections, read and write request volume, transaction success rate, and instance load (collection frequency: 500ms).

[0030] It adopts a multi-protocol universal data acquisition interface: supporting mainstream data acquisition protocols such as SNMP, JDBC, Prometheus, SSH, and Telnet, and adapting to different brands of servers, network devices, and databases; it is designed with a data cleaning and standardization module: deduplicating, completing, and formatting the raw data, and outputting it as structured standard data in a unified format such as JSON and XML, including but not limited to data identifiers, acquisition time, indicator values, and unique device IDs.

[0031] A distributed data acquisition architecture is adopted: multiple acquisition nodes are deployed within a private cloud cluster to achieve distributed data acquisition and redundant backup, avoiding single points of failure of acquisition nodes; the acquisition frequency can be dynamically configured: the default acquisition frequency is 500ms, and it can be adjusted to 100ms-10s according to business scenarios.

[0032] The core function of the integrated topology layer is to construct a server-network-database full-link dependency topology model based on standardized data from the data acquisition layer, and to realize dynamic updates and visualization of the topology.

[0033] The topology model includes three dimensions: physical dimension (the physical connection relationship between server nodes and network devices), network dimension (the logical connection relationship of network links, routing, load balancing, etc.), and application dimension (the binding relationship between database instances and server nodes, and network access policies).

[0034] Design a dynamic topology update mechanism: When devices are added / deleted, configurations are changed, or connection relationships are altered, the topology model is automatically updated with an update delay of ≤1 second.

[0035] Provides a topology visualization interface: It supports the graphical display of the entire chain of dependencies, and also provides an API interface for external systems to call.

[0036] The core functions of the intelligent diagnostic layer are as follows: As the core decision-making unit of the system, it realizes real-time detection of anomalies in servers, networks and databases in all dimensions, accurate determination of fault types, root cause tracing and identification of the scope of impact, and provides decision-making basis for linkage self-healing and traffic scheduling.

[0037] Three-layer technical architecture: This layer can adopt a three-layer architecture of multi-dimensional anomaly detection algorithm combination + fault rule engine + root cause localization reasoning model, as detailed below:

[0038] Multi-dimensional anomaly detection algorithm combination:

[0039] Timing indicator detection: Algorithms are used, such as sliding window (window size configurable: 5-60s) + 3σ principle + EWMA exponentially weighted moving average algorithm, to detect anomalies in timing indicators such as CPU utilization, link latency, and synchronization latency, with an accuracy rate of ≥99%.

[0040] Status indicator detection: A rule matching + heartbeat timeout detection algorithm (heartbeat timeout threshold: 1s) is used to detect anomalies in status indicators such as server heartbeat, network card status, and database instance status;

[0041] Correlation indicator detection: Using algorithms, such as Pearson correlation coefficient (correlation coefficient threshold: 0.8-0.95 configurable) + causal inference algorithm, correlation analysis is performed on cross-level correlation indicators (such as server hard disk failure and database IO anomaly) to filter false alarms with a false alarm rate of ≤0.1%.

[0042] The fault rule engine builds a custom rule library, which includes four major categories of fault rules: server, network, database, and linkage. Each category of rules contains three core elements: indicator threshold, abnormal duration, and correlation conditions.

[0043] A three-tier priority determination mechanism is designed: based on the scope of the fault impact (single instance / single node / cluster level), business importance (core business / non-core business), and fault urgency (immediate self-healing / delayed self-healing / manual intervention), three priority levels are set. Level 1 faults are cluster-level core business faults, triggering immediate self-healing; Level 2 faults are single-node non-core business faults, triggering delayed self-healing; Level 3 faults are single-instance minor faults, only triggering alarms without self-healing. The rule base supports visual configuration and dynamic updates without requiring a system restart.

[0044] Root cause localization inference model input parameters: anomaly detection results, fault rule matching results, integrated topology model, historical fault data; inference process: based on the dependency relationship of the topology model, starting from the anomaly indicators, tracing back to related devices / instances, and filtering secondary anomalies by combining causal inference algorithms to locate the root cause of the fault; output results: fault root cause type, unique ID of fault source device / instance, fault impact range, fault priority, and suggested self-healing strategy.

[0045] The core functions of the self-healing and traffic scheduling layer: As the core execution layer of the system, it realizes the atomic collaborative execution of database scheduling, network configuration updates, and traffic scheduling, which is the core innovation layer of this invention.

[0046] Atomic execution mechanism: Introducing distributed transaction locks and synchronization trigger signals to bind database scheduling, network configuration updates, and traffic scheduling operations into an atomic transaction, ensuring "either all succeed or all rollback"; Transaction timeout threshold: 5s, triggering automatic rollback if timeout occurs.

[0047] Database scheduling module: Supports two core operations: automatic master-slave switching and automatic instance migration, and automatically selects the operation type based on the root cause location results; Migration strategy: Selects the optimal healthy node based on the server node's hardware load, remaining resources, and network topology.

[0048] Network configuration update module: Executes synchronously with database scheduling operations, automatically updating routing policies, load balancing configurations, security group rules, and ACL access control lists to ensure that network configuration matches the location / role of the database instance.

[0049] Dynamic traffic scheduling module scheduling trigger conditions: fault self-healing trigger, database instance load exceeding threshold (configurable), network link latency / packet loss rate exceeding threshold (configurable), server node hardware load exceeding threshold (configurable).

[0050] Scheduling strategies: ① During fault self-healing: Seamlessly switch business traffic to healthy database instances / server nodes; ② During load skew: Achieve intelligent traffic routing and load balancing based on a multi-dimensional weighting algorithm of "database instance load + server hardware load + network link quality"; ③ During primary / standby failover: Switch write traffic to the primary database and distribute read traffic evenly to the standby database to achieve read / write separation optimization; Scheduling execution method: Implemented by updating the network layer load balancing configuration and routing strategy, with an execution latency of ≤2s.

[0051] The core functions of the verification and audit layer are: to realize automated validity verification after self-healing and traffic scheduling operations, full-process log recording, and automatic rollback of anomalies, forming a closed loop of operation and maintenance.

[0052] The automated verification module verifies the following metrics: database service availability (whether it can connect normally and read / write), network connectivity (link latency, packet loss rate), traffic scheduling effectiveness (whether traffic is forwarded correctly and without skew), and configuration consistency (whether the database configuration matches the network configuration).

[0053] Verification method: Simulate business requests to perform read / write tests, network link detection, and traffic monitoring; verification timeout threshold: 3 seconds; number of verification attempts: 3 (failure will trigger rollback).

[0054] Automatic rollback module rollback trigger conditions: verification failure, atomic transaction timeout, and anomaly during operation execution; Rollback strategy: adopts an atomic rollback mechanism to restore the database configuration, network configuration, and traffic scheduling strategy to the state before the operation, with a rollback execution delay of ≤2s; after the rollback is completed, the fault is re-detected, and if it still exists, an alarm is triggered to notify the operation and maintenance personnel.

[0055] The audit log module records the following information: fault information (root cause, type, priority), operation content (specific parameters for database scheduling, network configuration updates, and traffic scheduling), operation time, execution result, verification result, and rollback information (if any). Log storage strategy: 90-day rolling storage, supports retrieval by device ID, fault type, and time, and provides log export functionality to meet compliance and internal control requirements.

[0056] like Figure 2 As shown, the core process of this invention is a closed-loop process of "data acquisition → topology construction → intelligent diagnosis → fault decision → linkage self-healing and traffic scheduling → automated verification → result processing → continuous monitoring". The entire process requires no manual intervention, and the fault recovery time is ≤10s. The specific steps are as follows:

[0057] Step 1: Data Collection;

[0058] The data acquisition layer uses a multi-protocol universal acquisition interface (supporting SNMP, JDBC, Prometheus, SSH, Telnet, etc.) to collect real-time, multi-dimensional status data from servers, network devices, and database clusters. After cleaning and standardization, the data is uniformly output as structured data in JSON / XML format and transmitted to the integrated topology layer and intelligent diagnostic layer. The default acquisition frequency is 500ms, which can be dynamically adjusted according to business scenarios (100ms-10s).

[0059] Step 2: Topology Construction—Full-Link Dependency Topology Model;

[0060] This step constructs a multi-dimensional dynamic heterogeneous graph, using "devices / instances" as nodes and "physical connections," "network logical dependencies," and "application deployment / replication" as edges. The model specifically comprises three layers: physical layer topology, network layer topology, and application layer topology. The physical layer topology uses servers, switches, and routers as nodes, describing physical connections through edges of type "CONNECTS_TO" (e.g., server A's network interface card 1 connects to port Gig0 / 1 of switch B). The network layer topology uses load balancers, routing tables, and ACL rules as nodes, describing logical connections through edges of types such as "ROUTED_TO" and "BOUND_TO" (e.g., the backend pool of load balancer VS1 contains the IP address of server A). The application layer topology uses database instances (primary / backup) as nodes, describing application deployment and data replication relationships through edges of types such as "DEPLOYED_ON" and "REPLICATES_TO" (e.g., primary database DB01 is deployed on server A, and backup database DB02 synchronizes data from primary database DB01).

[0061] The topology implementation is accomplished collaboratively by the following four sub-modules: The resource discovery and registration module, which scans pre-configured IP address ranges during system startup or periodically, probing network devices, database instances, and servers via protocols such as SNMP / SSH, and outputting a raw resource list containing unique device IDs and basic attributes. The relationship detection and resolution module establishes physical "server-switch" connections by reading LLDP information or address forwarding tables from switches; it resolves network logical relationships such as routing policies and load balancing pool members by logging into network devices via SSH and executing commands or calling REST APIs; it obtains master-slave relationships and replication topology by querying database system tables, and establishes deployment relationships through operating system process information. The topology graph construction and storage module uses a graph database to store the topology, storing node attributes and edge types and attributes; when relationship changes are detected, it generates incremental update statements and submits them in real time. The topology visualization and query interface provides a RESTful API for external querying of dependencies and a Web UI to render graph data into interactive force-directed graphs or hierarchical graphs. The dynamic update latency of the entire topology model is controlled within 1 second.

[0062] Step 3: Intelligent Diagnosis—Fault Detection and Root Cause Localization through Multi-Algorithm Fusion;

[0063] The intelligent diagnostic layer adopts a four-step pipeline architecture of anomaly detection → rule matching → root cause localization → impact analysis to perform full-dimensional diagnosis on the collected real-time streaming data, with a processing latency of ≤2s.

[0064] The anomaly detection module employs three core algorithms for different indicator types. The first type is time-series indicator detection, applicable to indicators such as CPU utilization, link latency, and master-slave synchronization latency. This algorithm combines a sliding window (window size configurable from 5 to 60 seconds), the exponentially weighted moving average (EWMA) method, and the 3σ principle. It calculates the weighted moving average of the indicator within the sliding window and compares it with the actual value to obtain the residual. When the residual consistently exceeds three times the standard deviation, it is considered an anomaly. The second type is status indicator detection, applicable to indicators such as heartbeat signals and network card status. This algorithm uses a finite state machine in conjunction with a timeout counter, setting the heartbeat timeout threshold to 1 second. When the number of consecutive timeouts reaches 3 (cumulative 3 seconds), the state machine transitions the target state to "anomaly." The third type is correlation indicator detection, suitable for analyzing the causal relationship between cross-level indicators such as hard disk failure and database I / O anomalies. This algorithm calculates the Pearson correlation coefficient between two indicator sequences within a 10-second window. When the absolute value of the correlation coefficient exceeds a preset threshold (e.g., 0.85) and there is a significant time lag, it is considered a strong causal association.

[0065] The fault rule engine compiles custom rules into decision trees or Rete networks and stores them in memory. The rule base includes four main categories: server, network, database, and linkage. Each rule defines metric thresholds, anomaly duration, and associated conditions. Priority is determined based on a comprehensive evaluation of three dimensions: the scope of the fault's impact, its business importance, and its urgency. The specific classification criteria are as follows: Level 1 faults are cluster-level core business faults that require immediate self-healing; Level 2 faults are node-level non-core business faults that can be re-determined after a 10-second delay; Level 3 faults are instance-level general business faults that only log and notify operations personnel, without triggering self-healing.

[0066] The root cause localization inference model takes anomaly detection results, fault rule matching results, an integrated topology model, and a historical fault mode library as input. Its inference process is as follows: First, each detected anomalous event (e.g., "database IO anomaly") serves as the starting point for inference. Then, it traverses backward along the "dependent" direction in the integrated topology graph. A typical tracing path is from the database IO anomaly to the server where it is deployed, and then to the server's hard drive. If a clear anomaly (e.g., hard drive failure) is found in the upstream node, all downstream anomalies (database IO anomaly, network latency anomaly) are marked as "secondary anomalies" and are not output as root causes. Finally, the model outputs a unique ID of the root cause node, the fault root cause type (e.g., SERVER_DISK_FAIL), the fault priority, a list of affected areas, and suggested self-healing strategies (e.g., [MIGRATE_DB, UPDATE_NET_CONFIG, SWITCH_TRAFFIC]).

[0067] Step 4: Fault Decision Making—Decision Orchestrator;

[0068] Fault decisions are made by a separate "decision orchestrator" module, which makes decisions based on finite state machines and policy tables, rather than simple if-else logic.

[0069] Input: The "Root Cause Location Result" object output by the intelligent diagnostic layer, which includes fault_type, severity, affected_entities, impact_scope (cluster level / node level / instance level), and suggested_actions (suggested self-healing action sequence).

[0070] The decision-making logic process is as follows:

[0071] First, a linkage determination is performed. The system checks whether the fault type exists in the preset "Linkage Fault Type Table," such as a linkage fault between the server and the database, or a linkage fault between the network and the database. If it exists, the linkage self-healing process is initiated; otherwise, the process switches to a normal alarm or single-layer processing flow.

[0072] Next, the system determines whether self-healing will be triggered. The system makes a decision based on the severity level and impact scope of the fault, combined with the policy configuration table. The decision rules are as follows: when the severity level is Level 1 and the impact scope is at the cluster level, the decision is "trigger immediate self-healing," achieving a zero-latency response; when the severity level is Level 2 and the impact scope is at the node level, the decision is "delay for 10 seconds and re-determine" to observe whether the fault can recover automatically; when the severity level is Level 3 and the impact scope is at the instance level, the decision is "only record the alarm, do not trigger self-healing," requiring manual intervention; for other combinations, the system executes according to the configurations defined by the operations personnel in the rule base.

[0073] Finally, a strategy combination is selected. The system searches the "fault-action mapping table" based on the fault type to determine the specific self-healing action sequence. Typical mapping relationships include: for server hard disk failure (fault code SERVER_DISK_FAIL), the corresponding self-healing action sequence is instance migration, network configuration update, and traffic switching; for network link jitter (fault code NETWORK_JITTER), the corresponding self-healing action sequence is updating network routing and updating load balancing configuration; for database master database failure (fault code DB_MASTER_DOWN), the corresponding self-healing action sequence is database failover, updating network access control lists, and updating traffic scheduling policies; for database load skew (fault code DB_LOAD_SKEW), the corresponding self-healing action sequence is adjusting load balancing weights and updating traffic scheduling policies.

[0074] Output: The decision orchestrator ultimately generates a "self-healing task object", which contains a unique task identifier, an ordered list of actions, a transaction timeout threshold (default 5 seconds), and a rollback strategy (default atomic rollback), and sends it to the linkage self-healing and traffic scheduling layer for execution.

[0075] Step 5: Linked self-healing and traffic scheduling – atomized collaborative execution;

[0076] The linkage self-healing and traffic scheduling layer adopts atomic executors. Based on the concept of distributed transactions (two-phase commit), it achieves atomic coordination of database scheduling, network configuration updates, and traffic scheduling through local transaction logs and distributed locks, with an execution latency of ≤5s.

[0077] The atomic execution mechanism consists of three phases. In the preparation phase, the system acquires distributed locks on the relevant database instances and network devices based on Redis's Redlock algorithm, and records configuration snapshots such as the current load balancer pool members, master database instance ID, and routing table entries to the local transaction log. If lock acquisition times out (5 seconds), the task fails directly. In the execution phase, the system calls each sub-module in a predefined sequence of actions: the database scheduling module calls the DM8 failover or instance migration interface; the network configuration update module connects to the device via Netconf or SSH protocol to execute configuration commands; and the dynamic traffic scheduling module dynamically adjusts the weights of the load balancer's backend pool. During this phase, any failure or timeout of any sub-action will trigger a rollback. In the commit or rollback phase, if all actions succeed, the transaction is committed and the distributed lock is released; if any fails, the rollback operation is executed in reverse order (first rollback the network configuration, then rollback the database scheduling). The entire rollback process has an execution delay of ≤2 seconds, and the system re-detects the fault after the rollback is complete.

[0078] The dynamic traffic scheduling module supports multiple scheduling strategies. In fault self-healing scenarios, the scheduling strategy resets the weight of the faulty node to 0, seamlessly switching all business traffic to healthy nodes. In load skew scenarios, a multi-dimensional weighted intelligent routing algorithm is activated. This algorithm integrates three dimensions: database instance load, server hardware load, and network link quality. The weight of each dimension is configurable (default is 1 / 3 for each), calculating the comprehensive weight of each service node and dynamically adjusting the weight of the load balancer's backend pool members accordingly. In master-slave failover scenarios, write traffic is 100% switched to the newly elected master database, while read traffic is evenly distributed to all healthy standby databases using a round-robin algorithm, achieving read-write separation optimization. All traffic scheduling operations are implemented by modifying the load balancer configuration and performing hot reloading, with an execution latency of ≤2 seconds.

[0079] Step 6: Automated verification;

[0080] The verification and audit layer performs multi-dimensional automated verification of the availability of database services, network connectivity, traffic scheduling effectiveness, and configuration consistency after the operation.

[0081] The verification metrics for database service availability include the ability to connect normally and execute read and write operations. Verification is performed by simulating business requests to conduct SQL read and write tests. The normal threshold requires a 100% connection success rate and read / write latency not exceeding 100 milliseconds. Network connectivity verification metrics are link latency and packet loss rate, detected through ICMP or TCP Ping. The normal threshold requires latency not exceeding 50 milliseconds and a packet loss rate not exceeding 0.1%. Traffic scheduling effectiveness verification metrics are whether traffic is forwarded correctly and without skew, verified through traffic monitoring and comparison of distribution before and after forwarding. The normal threshold requires zero traffic on faulty nodes and a traffic distribution deviation of no more than 20% between healthy nodes. Configuration consistency verification metrics are whether the database configuration matches the network configuration, verified by comparing the database master IP with the load balancer backend pool IP list. A perfect match is required. The above verification process is executed three times, with a timeout threshold of 3 seconds for each verification. Failure in any verification initiates a rollback process.

[0082] Step 7: Result Processing;

[0083] The system performs differentiated processing based on the verification results. If the verification result is normal, the self-healing and traffic scheduling process is completed, and the audit log records the fault information, operation content, operation time, and execution result. The system then returns to step 1 for continuous monitoring. If the verification result is abnormal, an atomic automatic rollback mechanism is triggered. In addition to recording the above information, the audit log also records the failure reason and rollback information. After the rollback is completed, the fault is re-detected. If the fault still exists, an alarm is triggered to notify the operations and maintenance personnel. Alarm methods support SMS, email, and platform notifications, which can be selected according to the configuration.

[0084] Step 8: Continuous monitoring;

[0085] The system continuously collects status data from the server, network, and database layers, and detects anomalies and traffic scheduling triggers in real time, enabling 24 / 7 unattended intelligent operation and maintenance. The entire closed-loop cycle (from the occurrence of a fault to the completion of recovery verification) is ≤10 seconds.

[0086] like Figure 3 The diagram shown is a server-network-database end-to-end dependency topology model, illustrating the relationships between the physical, network, and application dimensions of the topology. It clearly presents the physical connections and logical bindings between server nodes, network devices, and database instances, providing topology support for root cause analysis and self-healing scheduling. While the topology model diagram illustrates these three dimensions, actual application scenarios are not limited to architectures such as 3 servers, 1 switch and router, 1 load balancer, or 1 primary and 2 backup databases.

[0087] The technical solution of the present invention will be described in detail below with reference to three typical application scenarios. This embodiment is only used to explain the present invention and is not intended to limit the scope of protection of the present invention. The test environment of this embodiment is: a private cloud database cluster of a financial enterprise, using the DM8 database, with 10 physical servers, 5 switches and 1 load balancer deployed to carry core transaction business, requiring 24 / 7 high availability and fault recovery time ≤10 seconds.

[0088] Example 1: Self-healing and traffic scheduling of interconnected faults caused by server hardware failure

[0089] Fault scenario: A physical server (ID: SRV01) in the cluster experiences a hard disk failure (health level ≤ 50%), and two database instances (ID: DB01, DB02) deployed on this server experience IO anomalies, which meet the conditions for a first-level linkage fault, triggering immediate self-healing and traffic scheduling.

[0090] Implementation process:

[0091] The data acquisition layer collected SRV01's hard disk health index at a frequency of 500ms, which was 45%, and DB01 / DB02's IO latency exceeded the threshold (200ms). After cleaning and standardization, the data was transmitted to the intelligent diagnostic layer and the integrated topology layer.

[0092] Based on the collected data, the integrated topology layer determines the binding relationship between SRV01 and DB01 / DB02, as well as the network links and load balancing strategies corresponding to DB01 / DB02;

[0093] The intelligent diagnostic layer uses "heartbeat timeout detection + Pearson correlation coefficient + causal inference algorithm" to locate the root cause of the fault as SRV01 hardware failure, and determines it to be a level 1 fault of server-database linkage, with an impact scope of single node level. The recommended self-healing strategy is "instance migration + network configuration update + traffic switching".

[0094] The system identifies the fault as a linkage-related fault and triggers the linkage self-healing and traffic scheduling process.

[0095] The self-healing and traffic scheduling layer initiates an atomic execution mechanism: ① migrate DB01 / DB02 to the healthy server node SRV05 (lowest load, sufficient resources) within the cluster; ② synchronously update the network-side load balancing configuration and routing policy, switching traffic originally directed to SRV01 to SRV05; ③ based on the read-write separation strategy, switch write traffic of DB01 / DB02 to the primary database and distribute read traffic to the backup database; the entire atomic transaction execution time is 3 seconds.

[0096] The verification audit layer performed connection tests and read / write tests on DB01 / DB02, detected network latency / packet loss rate, and monitored traffic forwarding. All three verifications were normal.

[0097] The audit log module records the entire process information, including fault information, migration process, configuration changes, and traffic scheduling, and the self-healing process is completed.

[0098] The entire process required no manual intervention, the fault recovery time was 8 seconds, the service was uninterrupted, and traffic forwarding was normal.

[0099] Example 2: Self-healing and traffic scheduling of interconnected faults caused by network jitter

[0100] Fault scenario: A network link (ID: LINK03) within the cluster experiences jitter, with a packet loss rate of 15% and a latency of 300ms, causing abnormal access to the database instance (ID: DB06) connected to this link. This meets the conditions for a first-level linkage fault, triggering immediate self-healing and traffic scheduling.

[0101] Implementation process:

[0102] The data acquisition layer collected data showing that LINK03 had a packet loss rate of 15% and a latency of 300ms, while DB06 had a connection failure rate of 20%. After standardization, the data was transmitted to each layer.

[0103] The integrated topology layer defines the logical connection relationships between LINK03, DB06, and the load balancer.

[0104] The intelligent diagnostic layer located the root cause of the fault as network link jitter, and determined it to be a level 1 fault of network-database linkage. The recommended self-healing strategy is "traffic scheduling + network routing update".

[0105] The self-healing and traffic scheduling layer initiates an atomic execution mechanism: ① Updates the network routing policy, switching traffic destined for LINK03 to the backup link LINK08; ② Updates the load balancing configuration, evenly distributing the service traffic of DB06 to the backup link. The traffic scheduling execution time is 2 seconds.

[0106] The audit layer verified that the packet loss rate (≤1%) and latency (≤50ms) of LINK08 and the connection failure rate (≤0.1%) of DB06 were normal.

[0107] The log records information throughout the entire process. The self-healing process is completed in 5 seconds, and there is no business interruption.

[0108] Example 3: Self-healing and traffic scheduling of linked faults caused by database master database failure

[0109] Fault scenario: The primary database (ID: DB09) in the cluster goes down, meeting the conditions for a first-level linkage fault, triggering primary / standby switchover, network configuration updates, and traffic scheduling.

[0110] Implementation process:

[0111] The data acquisition layer detected that DB09's heartbeat signal was lost, and after testing, it was confirmed that the main database was down.

[0112] The intelligent diagnostic layer located the root cause of the fault as the primary database crash, classifying it as a Level 1 fault involving the database, server, and network. The recommended self-healing strategy is "automatic primary / standby failover + network configuration update + read / write traffic scheduling".

[0113] The self-healing and traffic scheduling layer initiates an atomic execution mechanism: ① Switch the standby database DB10 to the primary database and update the primary and standby database configurations; ② Synchronously update the network-side security group / ACL rules and load balancing configurations, and bind them to the new primary database DB10; ③ Switch all write traffic to DB10 and distribute read traffic evenly to the remaining standby databases. The execution time is 4 seconds.

[0114] The audit layer verified the DB10's read / write functionality, network connectivity, and traffic forwarding effectiveness, and the verification was successful.

[0115] The log records information throughout the entire process. The self-healing process is completed in 7 seconds, and there is no business interruption.

[0116] Summary of Examples: The above three typical scenarios fully demonstrate that the present invention can effectively handle various interconnected faults of servers, networks, and databases, realize atomic self-healing and dynamic traffic scheduling, with fault recovery time ≤10s, seamless business recovery, no need for manual intervention, and meet the high availability operation and maintenance requirements of enterprise-level private cloud database clusters.

Claims

1. A server-network linkage fault self-healing and traffic scheduling system for private cloud database clusters, characterized in that, include: Data acquisition layer, integrated topology layer, intelligent diagnosis layer, linked self-healing and traffic scheduling layer, and verification and auditing layer; The data acquisition layer enables real-time, highly available, and standardized collection of full-dimensional status data from servers, network devices, and database clusters. The integrated topology layer, based on standardized data from the data acquisition layer, constructs a server-network-database full-link dependent topology model and enables dynamic updates and visualization of the topology. The intelligent diagnostic layer enables real-time detection of anomalies across all dimensions of servers, networks, and databases, accurate fault type determination, root cause tracing, and impact scope identification, providing decision-making basis for coordinated self-healing and traffic scheduling. The coordinated self-healing and traffic scheduling layer enables atomic collaborative execution of database scheduling, network configuration updates, and traffic scheduling. The verification and audit layer implements automated validity verification of self-healing and traffic scheduling operations, full-process log recording, and automatic rollback of anomalies, forming a closed loop of operation and maintenance.

2. The server-network linkage fault self-healing and traffic scheduling system for private cloud database clusters as described in claim 1, characterized in that, The data acquisition layer includes a multi-protocol universal acquisition interface and a data cleaning and standardization module. The multi-protocol universal acquisition interface supports mainstream acquisition protocols such as SNMP, JDBC, Prometheus, SSH, and Telnet, and is compatible with servers, network devices, and databases from different brands. The data cleaning and standardization module performs deduplication, completion, and format conversion on the acquired raw data, and outputs it as structured standard data.

3. The server-network linkage fault self-healing and traffic scheduling system for private cloud database clusters as described in claim 1, characterized in that, An integrated topology layer constructs a server-network-database end-to-end dependency topology model. The topology model includes three dimensions: the physical dimension, which is the physical connection relationship between server nodes and network devices; the network dimension, which is the logical connection relationship between network links, routing, load balancing, etc.; and the application dimension, which is the binding relationship between database instances and server nodes and network access policies. When devices are added / deleted, configurations are changed, or connection relationships are altered, the topology model is automatically updated with an update latency of ≤1 second. A topology visualization interface is provided: it supports graphical display of the end-to-end dependency relationship, and an API interface is also provided for external systems to call.

4. The server-network linkage fault self-healing and traffic scheduling system for private cloud database clusters as described in claim 1, characterized in that, The intelligent diagnostic layer includes a multi-dimensional anomaly detection module, a fault rule engine module, and a root cause localization inference model module. The multi-dimensional anomaly detection module uses a sliding window + 3σ principle + EWMA exponential weighted moving average algorithm to detect anomalies in time-series indicators; it uses a rule matching + heartbeat timeout detection algorithm to detect anomalies in server status indicators; and it uses a Pearson correlation coefficient + causal inference algorithm to perform correlation analysis on cross-level related indicators and filter false alarms.

5. The server-network linkage fault self-healing and traffic scheduling system for private cloud database clusters as described in claim 4, characterized in that, The fault rule engine module builds a custom rule library, which includes four categories of fault rules: server, network, database, and linkage. Each category of rules includes three core elements: indicator threshold, abnormal duration, and correlation conditions. A three-level priority determination mechanism is designed: three levels of priority are set according to the scope of fault impact, business importance, and fault urgency. Level 1 faults are cluster-level core business faults, which trigger immediate self-healing. Level 2 faults are single-node non-core business faults that trigger delayed self-healing. Level 3 faults are minor faults affecting a single instance, which only trigger an alert and do not self-heal.

6. The server-network linkage fault self-healing and traffic scheduling system for private cloud database clusters as described in claim 4, characterized in that, The root cause localization reasoning model module takes the following input parameters: anomaly detection results, fault rule matching results, integrated topology model, and historical fault data; based on the dependencies of the topology model, it starts from the anomaly indicators, traces back to related devices / instances, and combines causal reasoning algorithms to filter secondary anomalies and locate the root cause of the fault; the output results are: fault root cause type, unique ID of the fault source device / instance, fault impact range, fault priority, and suggested self-healing strategy.

7. The server-network linkage fault self-healing and traffic scheduling system for private cloud database clusters as described in claim 1, characterized in that, The integrated self-healing and traffic scheduling layer includes an atomic execution module, a database scheduling module, a network configuration update module, and a dynamic traffic scheduling module; The atomic execution module introduces distributed transaction locks and synchronization trigger signals to bind database scheduling, network configuration updates, and traffic scheduling operations into an atomic transaction, ensuring "either all succeed or all rollback"; The transaction timeout threshold is 5 seconds; if the timeout occurs, an automatic rollback will be triggered.

8. The server-network linkage fault self-healing and traffic scheduling system for private cloud database clusters as described in claim 7, characterized in that, The database scheduling module supports two core operations: automatic master-slave failover and automatic instance migration. It automatically selects the operation type based on the root cause analysis results. The migration strategy selects the optimal healthy node based on the server node's hardware load, remaining resources, and network topology. The network configuration update module executes synchronously with the database scheduling operations, automatically updating routing policies, load balancing configurations, security group rules, and ACL access control lists to ensure that the network configuration matches the location / role of the database instance.

9. The server-network linkage fault self-healing and traffic scheduling system for private cloud database clusters as described in claim 7, characterized in that, The dynamic traffic scheduling module seamlessly switches business traffic to healthy database instances / server nodes during fault self-healing. When load is skewed, it uses a multi-dimensional weighting algorithm based on "database instance load + server hardware load + network link quality," considering the load conditions of three dimensions: the database instance's own load, the hardware load of the server node, and the quality of the network link. Each dimension is assigned a configurable weight coefficient and normalized to between 0 and 1. The final service node weight is calculated using the formula Weight = w1*(1 - DB_load) + w2*(1 - CPU_usage) + w3*(1 - latency / max_latency). Based on this weight, the system dynamically adjusts the traffic distribution ratio of each node in the load balancer's backend pool, achieving intelligent traffic routing. During primary / standby failover, the system switches 100% of write traffic to the newly elected primary database and distributes read traffic evenly to all healthy standby databases using a round-robin algorithm, achieving read / write separation optimization.

10. The server-network linkage fault self-healing and traffic scheduling system for private cloud database clusters as described in claim 1, characterized in that, The verification and audit layer includes an automated verification module, an automatic rollback module, and an audit log module. The automated verification module simulates business requests to perform read / write tests, network link probing, and traffic monitoring. The verification timeout threshold is 3 seconds, and the number of verification attempts is 3. If the verification fails, a rollback is triggered. The automatic rollback module uses an atomic rollback mechanism to restore the database configuration, network configuration, and traffic scheduling policy to their pre-operation state, with a rollback execution delay of ≤2 seconds. After the rollback is completed, the system re-detects the fault. If the fault still exists, an alarm is triggered to notify the operations and maintenance personnel. The audit log module stores data on a rolling basis for 90 days, supports retrieval by device ID, fault type, and time, and also provides log export functionality to meet compliance and internal control requirements.

Images