Self-healing method, device and equipment of stateful data system and storage medium

Abstract

The present disclosure provides a self-healing method and device of a stateful data system, equipment and a storage medium, relates to the technical field of computers, in particular to the technical field of cloud computing, big data, distributed systems, cluster systems, data disaster recovery and the like. The specific implementation scheme is as follows: receiving abnormal sensing information of a stateful data system; obtaining a stateful context in the stateful data system according to the abnormal sensing information; determining a current scene according to the abnormal sensing information; calling a scene decision maker corresponding to the current scene to obtain a scene decision result according to the stateful context; and dynamically selecting a self-healing strategy in the case that an abnormal node is determined according to the scene decision result.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the fields of cloud computing, big data, distributed systems, cluster systems, and data disaster recovery. Background Technology

[0002] With the development of cloud computing and big data, distributed storage systems have become crucial infrastructure. A distributed storage system is a stateful data system. Nodes in a stateful data system can locally persist business data, replicas, metadata, etc. Request processing typically relies on historical data retained by the nodes and the cluster state. Distributed storage and databases are commonly examples of this type. Currently, most stateful data systems, including distributed storage systems, rely on local node metrics or heartbeats to detect node failures. This approach offers limited fault detection dimensions and creates numerous blind spots. Summary of the Invention

[0003] This disclosure provides a self-healing method, apparatus, device, and storage medium for a stateful data system.

[0004] According to one aspect of this disclosure, a self-healing method for a stateful data system is provided, comprising: Receive anomaly detection information from stateful data systems; Based on the anomaly detection information, obtain the stateful context in the stateful data system; Based on the anomaly detection information, the current scene is determined; Invoke the scene decision-maker corresponding to the current scene, and obtain the scene decision result based on the stateful context; If an abnormal node is identified based on the scenario decision results, a self-healing strategy is dynamically selected.

[0005] According to another aspect of this disclosure, a self-healing device for a stateful data system is provided, comprising: The receiving module is used to receive anomaly detection information from stateful data systems; The acquisition module is used to acquire the stateful context in the stateful data system based on the anomaly perception information. The determination module is used to determine the current scene based on the anomaly perception information; The invocation module is used to invoke the scene decision-maker corresponding to the current scene and obtain the scene decision result based on the stateful context. The selection module is used to dynamically select a self-healing strategy when an abnormal node is identified based on the scenario decision results.

[0006] According to another aspect of this disclosure, an electronic device is provided, comprising: At least one processor; and The memory is communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described in the present disclosure.

[0007] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of this disclosure.

[0008] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of this disclosure.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a flowchart illustrating a self-healing method for a stateful data system according to an embodiment of the present disclosure. Figure 2 This is a schematic diagram illustrating the interaction between a self-healing system and a distributed system according to embodiments of the present disclosure; Figure 3 This is a flowchart illustrating a self-healing method for a stateful data system according to an embodiment of the present disclosure. Figure 4 This is a flowchart illustrating a self-healing method for a stateful data system according to an embodiment of the present disclosure. Figure 5 This is a flowchart illustrating a self-healing method for a stateful data system according to an embodiment of the present disclosure. Figure 6 This is a flowchart illustrating a self-healing method for a stateful data system according to an embodiment of the present disclosure. Figure 7 This is a schematic diagram of the structure of a multi-dimensional sensing intelligent fault self-healing closed-loop architecture for a stateful distributed storage system according to an embodiment of the present disclosure. Figure 8 This is a schematic diagram of the interaction process between the fault self-healing system and the distributed storage system according to an embodiment of the present disclosure; Figure 9 This is a schematic diagram of the structure of a self-healing device for a stateful data system according to an embodiment of the present disclosure; Figure 10 This is a schematic diagram of the structure of a self-healing device for a stateful data system according to an embodiment of the present disclosure; Figure 11 This is a block diagram of an electronic device used to implement embodiments of the present disclosure. Detailed Implementation

[0011] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0012] Distributed storage systems have blind spots in fault detection, which may only detect local anomalies within the storage nodes themselves, failing to cover more fault scenarios such as business access failures, replication group master-slave synchronization anomalies, and Software Development Kit (SDK) topology version conflicts. The large scale of distributed storage service instances can lead to a vicious cycle of storage alarms: frequent business problems -> passively increasing alarms -> struggling to respond to high-frequency alarms leading to decreased processing quality -> forcibly reducing alarms -> operations and maintenance personnel missing business problems -> even more frequent business problems.

[0013] In a distributed storage system, the same external alarm could be caused by hardware failure, network anomaly, resource contention, excessive overall business writes, or internal anomalies such as service anomalies, making it difficult to distinguish accurately.

[0014] Distributed storage systems are typically stateful data systems (or stateful services), where nodes carry real data replicas. If a simple "go offline / rebuild when an error occurs" approach is adopted, it may lead to problems such as insufficient replicas, data inaccessibility, leader election storms, and refactoring storms.

[0015] In a distributed storage system, if a storage node fails, it will not only affect a single machine, but may also cause one or more of the following problems: a sharp decline in the data read and write capabilities of the entire replication group to which the node belongs; jitter due to high-frequency master-slave switching; network and disk pressure surges due to frequent data failures and refilling of replicas; and local problems spreading into cluster-level failures.

[0016] Figure 1 This is a flowchart illustrating a self-healing method 100 for a stateful data system according to an embodiment of the present disclosure. In one embodiment, the method may include: S110, Receive anomaly detection information from a stateful data system; S120. Based on the anomaly perception information, obtain the stateful context in the stateful data system; S130. Determine the current scene based on the anomaly perception information; S140. Invoke the scene decision-maker corresponding to the current scene, and obtain the scene decision result based on the stateful context; S150. If an abnormal node is identified based on the scenario decision results, a self-healing strategy is dynamically selected.

[0017] In this disclosure embodiment, see Figure 2 A stateful data system 201 can include one or more data clusters. A cluster can include multiple replication groups, and a replication group can include multiple data nodes (also called node instances, node processes, etc.). Multiple nodes within a replication group are replicas of each other (also called replica nodes or replica instances). Multiple replicas within the same replication group can store the same data shards. A replication group can distinguish between a leader node and followers. The leader node handles core read and write processing; all write requests are executed by the leader node. It is responsible for generating data change logs, allocating data versions, and synchronizing changes to all followers. It also maintains replication group metadata, handling shard routing, transaction commits, and data consistency checks. In the event of a leader failure, a re-election process can be initiated. Followers typically do not directly receive write requests but can continuously pull data logs synchronized by the leader node, replay updates to their local replicas, ensuring data consistency across multiple replicas; they can also handle read-only queries to distribute the load on the leader node.

[0018] In this disclosure embodiment, see Figure 2The data anomaly (or fault) self-healing (or damage mitigation) system may include a data layer and a control layer. The stateful data system 201 resides in the data layer, and the control layer can interact with the data layer to perform various controls. The control layer may include a cluster management service 202, a metadata management service 203, and an access status reporting center 204, and may also include a self-healing system 205. The cluster management service 202 is connected to the stateful data system 201, the metadata management service 203 is connected to the cluster management service 202 and / or the business terminal 206, and the access status reporting center 204 is connected to the business terminal 206. Any node within the stateful data system 201, the cluster management service 202, the metadata management service 203, or the access status reporting center 204, when detecting a potential anomaly within itself through polling, can proactively send anomaly detection information to the self-healing system 205, triggering a callback to the self-healing system to execute its self-healing method. This eliminates the need for the self-healing system to inspect each module or system; it only requires setting up interactive alarm interfaces or adjusting alarm triggering conditions. Therefore, it offers better compatibility and faster, more convenient anomaly detection. Consequently, the anomaly detection information sources for the self-healing system can be diverse, providing more dimensions of awareness compared to monitoring alarms on a single storage node, thus covering more anomaly scenarios (or fault scenarios, alarm scenarios, etc.). Compared to focusing only on local node anomalies, this upgrade moves to monitoring anomalies across the entire chain, including business processes, nodes, control plane, and topology consistency.

[0019] In some examples, the anomaly-sensing information of a self-healing system may include one or more of the following: Anomaly detection information from within the stateful data system, such as one or more of the following: slow request processing, request queue backlog, and insufficient disk space; Anomaly detection information from the access status reporting center, such as one or more of the following: request timeout, network failure causing the request to fail to reach the target node, and silent disconnection from the server. Anomaly detection information from the cluster management service, such as one or more of the following: heartbeat anomalies, master-slave synchronization gaps, and replica state differences; Anomaly-aware information from the metadata management service, such as one or more topology inconsistencies and topology update conflicts.

[0020] In this embodiment of the disclosure, after the self-healing system detects an anomaly, it may not directly process it, but instead complete the stateful context related to the anomaly. For example, the stateful context related to the anomaly may include: the basic environment state related to the anomaly detection information, the host hardware state, the location of the distributed storage topology, the replica state within the replication group, the storage engine type, and one or more of the master-slave roles.

[0021] For example, the basic environment status may include one or more of the following: Platform as a Service (PaaS) basic environment status, such as: PaaS agent status, Secure Shell (SSH) reachability, machine load, Transmission Control Protocol (TCP) retransmission rate, and the continuity of machine and instance monitoring data collection.

[0022] For example, host hardware status may include one or more of the following: Non-Volatile Memory Express (NVMe) wear leveling value, Solid State Drive (SSD) timeout completion event, and Central Processing Unit (CPU) model capabilities.

[0023] For example, the location of a distributed storage topology may include one or more of the following: cluster identifier and characteristics, replication group identifier and status, and replica instance set.

[0024] For example, the status of replicas within a replication group can include a comparison of the status of replicas within the replication group, specifically including: disk usage of each replica, read / write latency of each replica, data synchronization gap, and one or more of the total number of currently available replicas.

[0025] For example, storage engine types can include one or more of the following: object storage engine, block storage engine, file storage engine, key-value (KV) storage engine, etc.

[0026] By completing the stateful context, we can expand our focus from just one anomalous node to a multi-dimensional overall context that includes the anomalous node, the host machine, the replication group, the topology, and the role.

[0027] In this embodiment, based on multi-dimensional anomaly perception information, the anomaly scenario to which the current anomaly perception information belongs can be determined more accurately. The classification of anomaly scenarios can include various methods, such as: disk space scenarios, access availability scenarios, master-slave synchronization anomaly scenarios, instance anomaly scenarios, etc. Large scenarios can be further subdivided into smaller scenarios. For example, disk space scenarios can include insufficient disk space scenarios, access availability scenarios can include table access availability scenarios, and master-slave synchronization anomaly scenarios can include master-slave synchronization gap scenarios, etc. After determining the current scenario based on the multi-dimensional anomaly perception information, the scenario decision-maker corresponding to the current scenario can be invoked based on the previously collected stateful context and / or anomaly perception information. A scenario decision-maker can include decision-making process logic for the corresponding scenario. Invoking the scenario decision-maker corresponding to the current scenario can combine the previously collected stateful context and / or anomaly perception information to execute the decision process for the current scenario, obtaining the decision result for the current scenario. If an anomaly node can be clearly identified in the decision result, a self-healing strategy can be dynamically selected for that anomaly node.

[0028] According to embodiments of this disclosure, by comprehensively considering anomaly perception information and stateful context, a suitable scenario decision-maker can be invoked to obtain more accurate scenario decision results, thereby selecting a more appropriate self-healing strategy, which is beneficial to improving the stability of the stateful data system. Compared with schemes that only focus on a single node, the scenario selection, scenario decision results, and self-healing strategies are all more accurate due to the more comprehensive reference information, thus reducing error self-healing and effectively improving the stability of the stateful data system.

[0029] Figure 3 This is a flowchart illustrating a self-healing method 300 for a stateful data system according to an embodiment of this disclosure. This method can be used to implement one or more features of the self-healing method for the stateful data system described above. In one embodiment, the current scenario is a disk space scenario, and the determination of the disk space scenario includes: determining insufficient disk space based on anomaly perception information from the stateful data system.

[0030] In one implementation, the scenario decision-maker corresponding to the current scenario is invoked, and a scenario decision result is obtained based on the stateful context, including: S310. Call the disk space decision-maker corresponding to the disk space scenario to query the topological location of the alarm node in the stateful context; S320. Perform a horizontal comparison of replicas based on the topological location of the alarm node to obtain the horizontal comparison result.

[0031] In this embodiment, different current scenarios can be determined based on different anomaly perception information. If the monitoring platform of a state data system polls and finds that the remaining disk space of a certain data node is lower than a set threshold, such as 10% or 15%, it can be determined that the current scenario is a disk space scenario, or specifically a disk space shortage scenario. In this case, the disk space decision-maker can be invoked to make scenario decisions based on the alarmed data node (hereinafter referred to as the alarm node). The topology location of the alarm node N1 can be found from the previously collected distributed storage topology locations with state context, such as the number of the replication group to which the alarm node belongs, and the numbers of other replica nodes N2 and N3 included in the replication group. The alarm node and its replica results are compared horizontally to obtain the horizontal comparison results.

[0032] According to embodiments of this disclosure, by executing a dedicated process through a disk space decision-maker for disk space scenarios, it is beneficial to conduct a comprehensive analysis of the causes of insufficient disk space based on the stateful context, and obtain the accurate root cause of insufficient disk space.

[0033] In one implementation, performing a replica lateral comparison based on the topological location of the alarm node includes: Based on the topological location of the alarm node, the replication group to which the abnormal node belongs and all its replica nodes are identified; Analyze the disk usage status of all replica nodes within the replication group; If the disk usage of multiple replica nodes exceeds the threshold, it is determined that the overall write volume is too large, and a capacity expansion suggestion is output. If only the replica node represented by the alarm node exceeds the threshold, it is determined to be a single point of failure.

[0034] In this embodiment, all nodes within a replication group can be referred to as replica nodes, or simply replicas. After locating the alarm node N1 and its other replica nodes N2 and N3 within the replication group, the disk usage status of all replica nodes N1, N2, and N3 within the replication group can be analyzed to determine which specific replica node(s) has a disk usage status exceeding a threshold (e.g., higher than 90%). If the disk usage status of all replica nodes exceeds the threshold, it can be determined that the overall write volume is too high, and expansion suggestions can be output to the upper layer or a designated communication node without self-healing, proceeding to in-depth analysis. If only the disk usage status of the alarm node (represented by the replica node) N1 exceeds the threshold, it can be determined that N1 is a single point of failure.

[0035] In one implementation, prior to dynamically selecting a self-healing strategy, the method includes: S330. Call the general decision-maker to perform a health check on the abnormal node, and confirm again whether the abnormal node is abnormal based on the health check results, and / or the root cause of the abnormality in the current scenario.

[0036] In this embodiment of the disclosure, if the overall write volume is determined to be too large, or if a single point of failure is determined to be abnormal, in-depth analysis can be initiated. In-depth analysis may specifically include: invoking a general decision-maker, such as a general host environment health checker, to check whether the host machine of the abnormal node or alarm node N1 has one or more of the following problems: (1) Abnormal CPU load indicates that the root cause may be: contention for resources across different regions; (2) SSD wear exceeds limit / kernel log has timeout event, indicating that the root cause may be: hardware failure; (3) Other anomalies, such as an unhealthy Pass Agent, indicate that the root cause may be: an abnormal container environment; (4) If all are normal, the root cause may be: abnormal internal state of the storage service.

[0037] By comparing replicas horizontally, different disk problems can be distinguished and different decision-making suggestions can be given. For single-point anomalies, further analysis can be conducted on the abnormal node to obtain more accurate root causes and / or decision results.

[0038] Figure 4 This is a flowchart illustrating a self-healing method 400 for a stateful data system according to an embodiment of this disclosure. This method can be used to implement one or more features of the self-healing method for the stateful data system described above. In one embodiment, the current scenario is an access availability scenario, and the determination of the access availability scenario includes: determining, based on anomaly perception information from the access status reporting center, that the availability of the business access table in the data node does not meet a set threshold.

[0039] In this embodiment, the business-side access status reporting center can poll for one or more access status-related information, such as the business's own status, cluster metadata, read / write metrics, replica synchronization information, and resource status. For example, the business's own status may include one or more of the following: connection pool connection count, number of failed reconnections, read / write time, timeout errors, and request success rate. Cluster metadata may include one or more of the following: shard routing, master / slave instance addresses, replica online status, and shard migration progress. Read / write metrics may include one or more of the following: queries per second (QPS), read / write latency, batch operation throughput, and hotspot shard access frequency. Replica synchronization information may include one or more of the following: slave synchronization latency, log lag, and replica offline markers. Resource status may include one or more of the following: target instance disk utilization, input / output (IO) load, and network round-trip latency.

[0040] When the business SDK reads and writes tables in a stateful data system, it automatically reports the access status to the SDK access status reporting center periodically. This includes the number of read / write operations, the number of failures (such as flow control and timeout failures), the SDK business instance, the storage node machine (host machine) where the access failed, and the service port. When the SDK access status reporting center detects that the availability of the business access table cannot meet the preset value, such as 99.99%, it starts to notify the self-healing system to begin analyzing the anomaly (fault). For example, if the network card of the machine where a node is located fails, the corresponding node will continuously time out when the business reads and writes tables through the SDK. However, the node itself cannot detect that the business SDK has initiated a request and failed. In this case, after the self-healing system receives the anomaly detection information from the access status reporting center, such as a timeout of a target node (the storage table within it) or a network failure preventing the request from reaching the target node, it can determine that the availability of the business access table in this target node does not meet the set threshold, and thus identify it as a table access availability scenario.

[0041] In one implementation, the scenario decision-maker corresponding to the current scenario is invoked, and a scenario decision result is obtained based on the stateful context, including: S410. Call the availability decision-maker corresponding to the availability scenario, traverse the interfaces of each node instance of the business access status reporting center in the stateful context, and obtain the continuous sharded data maintained on each node instance. S420. Statistical analysis is performed on the access requests of the continuous fragmented data to obtain global statistical results; S430. If the access failure rate of one or more data nodes in the global statistics exceeds the first access threshold, the one or more data nodes shall be determined as abnormal nodes. S440. If there are no nodes in the global statistics results with a failure rate exceeding the first access threshold, filter out abnormal nodes with a failure rate exceeding the second access threshold, where the second access threshold is less than the first access threshold.

[0042] In one implementation, prior to dynamically selecting a self-healing strategy, the method includes: S450. Call the general decision-maker to perform a health check on the abnormal node, and confirm again whether the abnormal node is abnormal and / or the root cause of the abnormality in the current scenario based on the health check results.

[0043] Examples of the decision-making process of the availability decision-maker and / or general decision-maker in this disclosure embodiment are as follows: Level 1, Precise Analysis: The availability decision-maker iterates through the interfaces of each node instance in the request SDK access status reporting center, obtaining two consecutive shard data maintained on each node instance. Global statistics are performed, such as global normalized summation: the total number of requests is summed, and the counts of various read / write timeouts / failures are summed separately, to aggregate the proportion of failed storage nodes to the total number of requests. If the failure rate of a certain storage node exceeds a first access threshold (e.g., 70% or 80%), the result is considered directly reliable, and self-healing can continue, with self-healing operations executed through the execution layer.

[0044] The second level, fallback recall, involves filtering all nodes whose access failure rate exceeds a second access threshold (e.g., 5% or 10%). Each node undergoes secondary fault confirmation by calling a general decision-maker to analyze for anomalies such as hardware failures or machine environment malfunctions. If a secondary fault is confirmed, self-healing can proceed through the execution layer. Alternatively, the first level of precise analysis can be performed first, followed by the second level of fallback recall if the precise analysis fails to identify the anomalous node. Or, precise analysis and fallback recall can be performed in parallel, with self-healing applied to all anomalous nodes.

[0045] Level 3, SDK-assisted analysis: If neither precise analysis nor fallback recall reveals any node anomalies, the access timeout may be caused by a fault in the SDK service instance itself. The availability decision-maker can proactively pull instance runtime logs from the SDK access status reporting center, filter and aggregate the SDK service instances and their respective machines. It analyzes whether there is clustering of SDK service instances. If a certain service instance accounts for more than 60% of all failures, it is considered highly likely that the problem is caused by the SDK service instance. Node self-healing is not required; the analysis and decision results are automatically sent to the upper-level decision-making end.

[0046] According to embodiments of this disclosure, an availability decision maker can quickly and accurately locate the abnormal node causing the availability drop from a large number of data nodes in a stateful data system. For example, in a distributed storage system where table data is scattered across tens to thousands of storage nodes, this solution can accurately and quickly pinpoint whether the availability drop is caused by the failure of a small number of machines.

[0047] Figure 5 This is a flowchart illustrating a self-healing method 500 for a stateful data system according to an embodiment of this disclosure. This method can be used to implement one or more features of the self-healing method for the stateful data system described above. In one embodiment, the current scenario is a master-slave synchronization gap scenario, and the determination of the master-slave synchronization gap scenario includes: determining that the master-slave synchronization gap is greater than a set threshold based on anomaly perception information from a cluster management service.

[0048] In this embodiment, the underlying storage cluster of the engine is an active, automatic data synchronization cluster. This means that after business data is written to the master node (master role node) via the SDK, it is automatically synchronized to all slave nodes (slave role nodes). Each data write generates a monotonically increasing sequence number. Under normal circumstances, the data between the master and slave nodes should be synchronized quickly and promptly. However, if a slave node experiences hardware failure, network jitter, or disk performance degradation, causing log synchronization to lag, the difference between the latest data sequence numbers of the master and slave nodes may widen, preventing the business SDK from accessing the latest written data. Nodes periodically interact with the cluster management service via heartbeat requests, recording the latest data sequence number in the current replica within the heartbeat information. Upon receiving the sequence numbers of all master and slave nodes in the same replication group, if the cluster management service detects that a slave node's data sequence number lags behind the master node by a set threshold (e.g., 1000), it can send an anomaly detection message to the self-healing system, triggering a callback to notify the system to begin anomaly analysis.

[0049] In one implementation, the scenario decision-maker corresponding to the current scenario is invoked, and a scenario decision result is obtained based on the stateful context, including: S510. Call the master-slave synchronization gap decision-maker corresponding to the master-slave synchronization gap scenario, and obtain all master-slave nodes of the replication group whose master-slave synchronization gap is greater than the set threshold through the stateful context from the meta information management service. S520. Based on the latest sequence number query interface of the master-slave node, find the slave node whose difference from the master node reaches a threshold. The slave node is an abnormal node.

[0050] In one implementation, prior to dynamically selecting a self-healing strategy, the method includes: S530. Call the general decision-maker to perform a health check on the abnormal node, and confirm again whether the abnormal node is abnormal and / or the root cause of the abnormality in the current scenario based on the health check results.

[0051] In this embodiment, the decision-making process of the master-slave synchronization gap decision-maker and / or the general decision-maker may include: the master-slave synchronization gap decision-maker parses the cluster management service instance that issued the alarm, actively pulls the latest logs of the instance, and filters out logs related to excessive differences in the latest data sequence numbers of the master and slave nodes. From these logs, the cluster name, replication group number, master-slave replica information, and log gap values ​​are parsed. All master and slave nodes in the corresponding replication group are obtained through the metadata management service, and the node is actively requested to query the latest sequence number interface. Based on the current latest data sequence number, the slave nodes whose differences from the master node reach a threshold are identified. For the identified slave nodes with large differences from the master node, the general decision-maker is invoked for secondary fault confirmation to analyze whether the node has any anomalies, such as hardware failure or machine environment failure. If the secondary confirmation of fault indicates that only one slave node is faulty, the self-healing execution layer can continue to perform loss mitigation operations. If more than one slave node is faulty, a fault identification notification can be sent to the superior decision-making end, and the self-healing operation is not actively performed.

[0052] According to embodiments of this disclosure, a master-slave synchronization gap decision-maker can quickly and accurately locate abnormal nodes when there is a significant data gap between master and slave nodes. This reduces timeliness issues caused by accessing old data and improves access speed and success rate. For example, the number of replication groups in each cluster of a distributed storage system can range from several thousand to tens of thousands. The master-slave synchronization gap decision-maker can perceive the replication group to which a storage node belongs, which replicas it contains, and the latest data sequence number of each replica. It can accurately identify shards with significant master-slave data gaps and quickly identify faulty nodes, promptly removing them and restoring them to normal machines, thus avoiding timeliness issues related to business access to old data.

[0053] Figure 6 This is a flowchart illustrating a self-healing method 600 for a stateful data system according to an embodiment of this disclosure. This method can be used to implement one or more features of the self-healing method for the stateful data system described above. In one embodiment, the current scenario is an instance anomaly scenario, and the determination of the instance anomaly scenario includes: determining that a data node is anomaly based on the anomaly perception information from the data nodes of the stateful data system.

[0054] In this embodiment, the instance anomaly scenario refers to an anomaly scenario perceived by the node instance itself. Examples include excessively long read / write engine latency within the storage node, backlog of external request processing, and process failure to start. In the alarm system of a stateful data system, callback rules can be set. Once a specified operational metric of a node instance reaches a threshold, a callback can be sent to notify the self-healing system to begin anomaly analysis. A node instance can be understood as a process running on node hardware, such as a host machine, and is sometimes simply referred to as a node.

[0055] In one implementation, the scenario decision-maker corresponding to the current scenario is invoked, and a scenario decision result is obtained based on the stateful context, including: S610. Call the instance exception decision maker corresponding to the instance exception scenario. For the exception that the node cannot start, actively pull the instance exception log of the node that cannot start. S620. Analyze the root cause of the startup failure in the exception log; S630. For the underlying environmental problems in the root cause, output the analysis conclusion and initiate an instance task to migrate the data node to redistribute the data node's data on other clusters.

[0056] In one implementation, before dynamically selecting a self-healing strategy, the following steps are included: S640. Call the general decision-maker to perform a health check on the abnormal node, and confirm again whether the abnormal node is abnormal and / or the root cause of the abnormality in the current scenario based on the health check results.

[0057] Examples of the decision-making processes of instance-specific decision-makers and / or general decision-makers in this disclosure are as follows: For exceptions where a node instance (process) fails to start, the instance exception decision maker proactively pulls the corresponding instance exception logs. It analyzes the root cause of the startup failure in the logs, such as a port being occupied, the SSD sector size not meeting the storage engine's expectations, or failure to interact with the PaaS Agent. For such fundamental environmental issues, it promptly sends the analysis results to the upper-layer decision-making end and initiates a task to migrate the instance, allowing PaaS to reassign the instance to another cluster. For exceptions where engine read / write data takes too long, the general decision maker is invoked to prioritize analyzing whether there are hardware-level faults, such as SSD erase / write lifespan expiration, hardware errors, or issues with machine CPU load. If a fault is confirmed, self-healing can continue, with the self-healing operation executed through the execution layer. For exceptions involving a backlog of external request processing, the general decision maker is invoked to analyze the fault. If a fault is confirmed, self-healing can continue, with the self-healing operation executed through the execution layer.

[0058] According to embodiments of this disclosure, node anomalies manifest in various ways, ranging from SSH unreachability and monitoring failure at the operating system level, to hardware issues, and even resource contention-related problems such as CPU overload or network quality degradation. The instance anomaly decision-maker supports a multi-dimensional, comprehensive decision-making mechanism for instance-level anomalies, automatically performing root cause aggregation and remediation path selection.

[0059] Furthermore, by combining scenario-specific decision-makers with general decision-makers for attribution, it is beneficial to significantly reduce the misjudgment rate, improve attribution accuracy, and enhance the accuracy of self-healing operations based on the fusion analysis of multi-source heterogeneous data.

[0060] In one implementation, before dynamically selecting a self-healing strategy, one or more of the following security checks are performed: Check whether the number of available replicas in the replication group meets the threshold; replica nodes below the threshold are not allowed to be removed. Based on historical self-healing records, check whether the replica node represented by the abnormal node has been subjected to self-healing operations multiple times within a set time period in the past. If the frequency of self-healing operations exceeds the upper limit, replace the self-healing operation with a notification. Check if the system-level self-healing switch is enabled; if not, disable the global self-healing capability. Check if the self-healing switch of the current cluster is enabled; if not, disable the self-healing capability of the current cluster.

[0061] In this embodiment of the disclosure, multiple security mechanisms can be preset, and one or more security mechanisms can be executed before the self-healing operation is performed at the execution layer.

[0062] For example, replication group availability protection: check the number of available replicas in the replication group, and if the number of replicas is lower than the threshold, abnormal nodes are not allowed to be removed.

[0063] For example, self-healing operation frequency control: if the frequency of self-healing operations in the same data center / cluster exceeds the upper limit within a short period of time, it will automatically degrade to only notify the available loss prevention methods after identification.

[0064] For example, in system-level start / stop control: if the system-level self-healing switch is not enabled, self-healing is globally disabled, and the execution layer does not perform any self-healing operations. If the system-level self-healing switch is enabled, the execution layer performs self-healing operations normally.

[0065] For example, cluster-level start / stop control: checks whether the self-healing switch of the current cluster containing the abnormal node is enabled. If it is not enabled, the self-healing of the cluster is disabled, and the execution layer does not perform the self-healing operation for this cluster. If it is enabled, the execution layer performs the self-healing operation for this cluster normally.

[0066] For example, the over-frequency degradation notification mechanism: if too many anomalies are detected in a short period of time, there may be a large-scale failure. In this case, the automatic self-healing will no longer be performed, and the issue will be reported to the decision-making end for manual decision-making.

[0067] According to embodiments of this disclosure, by performing pre-checks through various security mechanisms, security safeguards can be added to the system before the self-healing operation is performed at the execution layer, preventing system crashes or other anomalies caused by self-healing.

[0068] In one implementation, the self-healing strategy is dynamically selected, including one or more of the following: If the abnormal node is the master node, a master-slave switch will be performed first. If the abnormal node is a slave node and the health status of the replica node represented by the abnormal node meets the availability requirements, the abnormal node is removed, and a new node is started to perform abnormal recovery of the new replica node. If the abnormal node is determined to be a hardware malfunction based on the health check results of the abnormal node, the instance of the abnormal node shall be expelled or deleted. If the disk usage of multiple replica nodes in a replication group exceeds the threshold, it is determined that the overall write volume is too large, and a capacity expansion suggestion is output. If the number of available replicas in the replication group is insufficient, an exception notification is output but no self-healing operation is performed; the fault is waited for recovery. If the environment is normal but the service is abnormal, suggest manual intervention.

[0069] In this embodiment, master and slave roles can be distinguished. If the abnormal node is the master node, a master-slave switch can be performed first. Within the replication group, a vote can be initiated through the consistency protocol to elect the slave node (slave instance) with the most complete data to be promoted to the new master node (master instance). The cluster updates the metadata to record the address of the new master node, and the remaining slave nodes switch their synchronization source to follow the new master node. The business SDK re-pulls the shard routes and forwards read and write requests to the master node. Finally, the replication group restores full read and write capabilities, and multiple replicas continue to synchronize, ensuring that data is not lost and that services are not interrupted for extended periods.

[0070] In this embodiment, if the faulty node is a slave node, although the replicas carried by the slave node can temporarily provide services and meet basic availability, there are potential faults within the slave node itself. Continuous operation is prone to sudden problems such as synchronization stalls and process crashes. Therefore, the old instance can be removed and a new replica can be created. This isolates the source of the fault and maintains a multi-replica redundancy architecture, ensuring long-term cluster stability. Specifically, the faulty instance can be marked offline in the cluster metadata, and the replica node represented by the slave node can be removed from the shard topology. A new instance can be launched on an idle host machine. The new instance can synchronize all data and incremental logs from the other healthy replicas to complete the replica reconstruction. The shard routing information can be updated, and the business SDK can be notified to switch the access target. In this way, the faulty instance, as a slave node, can be removed from the cluster and no longer bear traffic. The number of replicas in the replication group can also be restored to the preset redundancy standard, eliminating potential risks, maintaining data consistency and integrity, and preventing long-term business interruptions.

[0071] In this embodiment, if a hardware failure occurs, the cluster can first mark the abnormal node (instance) as offline, then automatically migrate all shard data on it to other healthy nodes (instances). After synchronization is complete, the cluster metadata is updated, and then the abnormal node (instance) is evicted or deleted from the cluster topology. The business SDK refreshes the shard routing, no longer forwarding requests to the abnormal node, and the number of cluster replicas is restored to the standard redundancy specification. In this way, the abnormal node can be removed from cluster control, the cluster returns to a healthy and balanced state, and business read and write operations are not continuously affected.

[0072] In this embodiment of the disclosure, if it is detected that the disk usage of multiple replica nodes in the replication group exceeds the threshold, it can be determined that the overall write volume is too large, and an expansion suggestion is output to the upper-level decision-making end. In this case, self-healing operations such as deletion and master-slave switching are not required for abnormal nodes.

[0073] In this embodiment of the disclosure, through a front-end security mechanism, if it is found that the number of available replicas in the replication group is insufficient, for example, there is only 1 replica, an abnormal notification can be output but no self-healing operation can be performed, waiting for fault recovery, or waiting for other instructions from the upper-level decision-making end.

[0074] In this embodiment, under normal underlying environment conditions such as hardware, network, disk, and server resources, if a node (instance) experiences service anomalies such as storage process crashes, internal thread freezes, replica synchronization logic errors, or port listening failures, manual intervention suggestions can be sent to the upper-level decision-making end to ensure system security. Manually checking logs can pinpoint program errors (bugs), misconfigurations, thread blocking, and other issues to prevent recurring failures.

[0075] According to the embodiments of this disclosure, it is able to identify master and slave roles, perceive the health status of replicas, perceive the topology location and characteristics, distinguish between expansion suggestions, replica removal, master-slave switching, instance migration, manual intervention and other actions, and distinguish between the processing of stateful and stateless data systems, making it more suitable for stateful data systems.

[0076] In one implementation, the method may further include: recording and / or pushing one or more of the following information: the root cause of the current scene anomaly, the topological location of the abnormal node, the self-healing operation performed on the abnormal node, and the self-healing result after performing the self-healing operation.

[0077] In this embodiment, the self-healing system may include an observability module for performing one or more of the following operations: pushing fault attribution conclusions; pushing the topology location of abnormal nodes; pushing execution actions and results; recording complete self-healing logs; and supporting one-click jump to the monitoring platform and availability report observation platform. By recording and / or pushing observable information, the self-healing process transforms from a "black box" into a "transparent closed loop," enabling full-link state observability. When an anomaly occurs, there is no need to guess the root cause layer by layer, significantly improving location efficiency. The entire process is traceable, facilitating rapid fault review and performance optimization, and reducing operation and maintenance costs.

[0078] In this disclosure, the stateful data system may include a stateful distributed storage system. An anomaly in the system can also be called a fault. Actions such as repair, remediation, deletion, and notification based on the anomaly (or fault) can be called self-healing. The stateful data system self-healing system in this disclosure can also be called a fault self-healing system. The architecture and working principle of the fault self-healing system are described below.

[0079] I. Overall Design Architecture of the Fault Self-Healing System Figure 7 This is a schematic diagram of a multi-dimensional sensing intelligent fault self-healing closed-loop architecture for a stateful distributed storage system according to an embodiment of this disclosure. It may include: a sensing layer 710, a decision layer 720, an execution layer 730, and an observability module 740. During execution at the execution layer 730, multi-layer security control can be implemented through a multi-layer security mechanism 750. This architecture has one or more of the following characteristics: 1. Fault diagnosis is not based solely on a single indicator, but rather on multi-dimensional perception, which integrates more stateful contexts to make fault diagnosis more accurate.

[0080] 2. It is not a simple rule judgment, but a joint attribution of multi-scenario decision-makers and general environmental health checkers.

[0081] 3. Instead of a one-size-fits-all approach to recovery, it uses dynamic self-healing based on a stateful context.

[0082] 4. Instead of executing directly, multiple layers of security controls are implemented first.

[0083] 5. It is not a black-box execution; the entire process is observable.

[0084] Internal design of the fault self-healing system II. Interaction Design of Fault Self-Healing System and Distributed Storage System Figure 8This is a schematic diagram illustrating the interaction process between a fault self-healing system and a distributed storage system according to an embodiment of this disclosure. It may include: the fault self-healing system 810 obtaining multi-dimensional perception information (e.g., anomaly information, fault information, or alarm information) from the storage nodes of the storage cluster of the distributed storage system 820 in the data layer, the SDK access status reporting center 830, the metadata management service 840, and the cluster management service 850. Then, the fault self-healing system completes the multi-dimensional context based on the multi-dimensional perception information, calls the decision-maker corresponding to the scenario in its own decision layer to obtain the decision result, and then selects the corresponding self-healing strategy based on the decision result. Before executing the self-healing strategy, a security mechanism can be checked, and the distributed storage system can be controlled to execute the self-healing strategy based on the security mechanism. Various information generated in the above process can be recorded through the observability module of the fault self-healing system.

[0085] The relationship between fault self-healing systems and distributed storage systems Third, the perception layer adopts an "event-driven + multi-dimensional perception" approach. The perception layer of the fault self-healing system in this embodiment can collaboratively perceive faults from four dimensions: 1. At the storage node level, at least one of the following fault data can be obtained: slow request processing; request queue backlog; insufficient disk space.

[0086] 2. At least one of the following fault data can be obtained from the business SDK access side: request timeout; network failure causing the request to fail to reach the node; server silently losing connection.

[0087] 3. At least one of the following fault data can be obtained from the cluster management service: abnormal heartbeat; master-slave synchronization gap; replica status difference.

[0088] 4. At least one of the following fault data can be obtained from the metadata management service dimension: the SDK holds a topology version that is inconsistent with the latest topology; topology update conflict.

[0089] According to the embodiments of this disclosure, it is possible to detect not only local anomalies of a single node, but also anomalies across the entire link, including services, nodes, control plane, and topology consistency, thereby improving the accuracy of fault detection.

[0090] IV. After the perception layer detects an anomaly, it can first complete the following stateful context: 1. At least one of the following can be completed in the Platform as a Service (PaaS) basic environment status dimension: PaaS agent status; Secure Shell (SSH) reachability; machine load; Transmission Control Protocol (TCP) retransmission rate; machine and instance monitoring and collection continuity.

[0091] 2. Complete at least one of the following in the host hardware status dimension: NVMe wear leveling value; SSD timeout completion event; CPU model capability.

[0092] 3. Complete at least one of the following in the distributed storage topology location dimension: cluster identifier and characteristics; replication group identifier and status; replica instance set.

[0093] 4. When comparing replica status within a replication group, complete at least one of the following dimensions: disk usage of each replica; read / write latency of each replica; data synchronization gap; and the total number of currently available replicas.

[0094] 5. Complete at least one of the following in the storage engine type and master / slave role dimensions: engine type; master / slave (Leader / Follower) role.

[0095] According to embodiments of this disclosure, when performing fault perception, it is possible to not only focus on a single point of failure, but also to obtain the overall context of the fault point, host machine, replication group, topology, and role, thereby improving the accuracy of fault perception and scenario decision-making.

[0096] The multidimensional contexts described above are merely examples and not limitations; they can be adjusted to other dimensions. For example: resource status dimension: CPU, memory, disk, network, etc.; business status dimension: Input / Output Operations Per Second (IOPS), latency, error rate, etc.; topology dimension: replica distribution, original relationships, etc.; time dimension: alarm time, duration, trend, etc.; history dimension: historical alarms, changes, fault records, etc.

[0097] Compared to focusing on only one node, the embodiments of this disclosure can focus on the overall context of multiple dimensions such as the point of failure, the host machine, the replication group, the topology, and the role, which is conducive to accurate subsequent decision-making.

[0098] V. The decision-making level adopts a combination of a "general-purpose health checker" and a "scenario-specific decision-making device". 1. General Host Environment Health Checker The General Host Environment Health Checker is a reusable basic judgment module that can perform multi-dimensional checks on the target host in the following dimensions: PaaS Agent health status; SSH reachability; CPU load; TCP retransmission rate; monitoring metric collection continuity; SSD wear level; SSD timeout events; host CPU model performance.

[0099] 2. Scenario-specific decision-makers: Different scenarios can call their own scenario-specific decision-makers. Here are some examples of scenario-specific decision-makers: Availability decision-maker; Master-slave synchronization gap decision-maker; Topology conflict decision-maker; Disk space decision-maker; Table access availability decision-maker; Instance exception decision-maker.

[0100] The perception dimension, context dimension, scene type, and decision-maker mentioned above are merely examples and not limitations. The specific choices can be made flexibly according to the needs of the actual application scenario.

[0101] Unlike the "single threshold + fixed action" approach, the embodiments of this disclosure achieve joint attribution by starting with scene recognition, making specific decisions, and then invoking general health checks when necessary.

[0102] VI. Dynamic stop-loss based on "stateful characteristics" at the execution layer Before executing an action, the execution layer of this disclosure will dynamically select the optimal stop-loss / self-healing strategy by comprehensively considering at least one of the following information: topology location and cluster characteristics; historical stop-loss (self-healing) records; storage engine type and characteristics; master-slave roles; and replica health status of the replication group.

[0103] An example of dynamically selecting the optimal strategy according to this disclosure is as follows: 1. In the event of a master node failure, a master-slave failover strategy should be prioritized; 2. In the event of a slave node failure, the removal strategy can be prioritized while ensuring replica availability. 3. When disk usage is high across all replicas, the recommended strategy of expanding output capacity should be prioritized; 4. In cases where the environment is normal but the service is abnormal, it is advisable to prioritize providing suggestions for manual intervention; 5. When the number of available replicas is insufficient, a strategy of only notifying without execution and waiting for the fault to be recovered can be prioritized.

[0104] According to the dynamic selection of the optimal strategy in the embodiments of this disclosure, "anomaly as action" can be upgraded to "understanding stateful semantics before taking action". After understanding the stateful semantics of the fault or alarm in the distributed storage system, the optimal strategy is selected based on the stateful semantics, thereby improving the effectiveness of self-healing or self-healing strategies.

[0105] VII. Multi-layered security mechanism An example of the multi-layered security mechanism in this disclosure is as follows: 1. Replication group availability protection: Replication groups cannot be removed if the number of available replicas falls below a threshold; 2. Self-healing operation frequency control: If the limit is exceeded in the same data center / cluster within a short period of time, it will automatically degrade to only notify the available loss prevention methods after identification; 3. System-level start / stop control: It can support disabling self-healing globally at any time; 4. Cluster-level start / stop control: Supports disabling self-healing for specified clusters; 5. Overly Frequent Degradation Notification Mechanism: In the event of too many anomalies or a large-scale failure in a short period of time, self-healing will no longer be performed automatically, and a degradation notification will be output. According to the multi-layered security mechanism of this disclosure, brakes and guardrails can be added to the self-healing system to make the overall self-healing process safe and controllable, and to protect the system from crashing when executing the self-healing strategy.

[0106] 8. The entire process is observable The observability module has one or more of the following functions, for example: 1. Push out fault attribution conclusions; 2. Push the topology location of abnormal nodes; 3. Push execution actions and results; 4. Keep a complete self-healing log; 5. Supports one-click jump to the monitoring platform and availability report observation platform.

[0107] The observability module transforms self-healing from a "black box" into a "transparent closed loop," enabling full-link state observability. When an anomaly occurs, there is no need to guess the root cause layer by layer, improving the efficiency of localization. The entire process is traceable, facilitating rapid review of faults and optimization of performance, and reducing operation and maintenance costs.

[0108] Here are some examples of decision-making systems for specific scenarios: Example 1: Fault self-healing in the case of insufficient disk space 1. Scenario Background: The replication group contains three replica nodes: N1, N2, and N3. The monitoring platform detected that the remaining disk space of N1 was less than 10%.

[0109] 2. The processing flow of this embodiment is as follows: Step 1: The perception layer receives an alarm, specifically a disk space alarm event from N1.

[0110] Step 2: The decision-making layer schedules the disk space decision-maker.

[0111] Step 3: Horizontal comparison of replicas. Query the topology location, identify the replication group it belongs to and all its replicas, and analyze the disk usage of all replicas N1, N2, and N3 within the same replication group: if N1, N2, and N3 all exceed the threshold, it is determined that the overall write volume is too large, and an expansion suggestion is output without self-healing.

[0112] If only N1 exceeds the threshold, it is judged as a single point of anomaly, and proceed to step 4 for in-depth analysis.

[0113] Step 4: In-depth analysis. Use the general host environment health checker to check the following metrics of the N1 host machine and determine the root cause, as shown in the example below: (1) In the case of abnormal CPU load, determine the root cause as mixed resource contention; (2) In the case of SSD wear exceeding the limit / kernel log having timeout events, determine the root cause as hardware failure; (3) In the presence of other anomalies, such as an unhealthy PaaS Agent, determine the root cause as an abnormal container environment; (4) If all indicators are normal, determine that the root cause is an abnormal internal state of the storage service.

[0114] Step 5: Security Verification. You can check the following: (1) Does the number of available replicas in the current replication group meet the threshold? (2) Has N1 been self-healed multiple times in the past hour? (3) Whether the system-level / cluster-level self-healing switch is turned on.

[0115] Step 6: Strategy Execution.

[0116] If N1 is a Follower and the replica health meets the requirements: execute node removal and start launching a new instance to perform new replica fault recovery.

[0117] If N1 is the Leader: prioritize master-slave switching, then consider subsequent actions; If it is a hardware failure: depending on the different abnormal situations, the PaaS platform will be triggered to evict or delete the instance to prevent the problem from recurring.

[0118] Step 7: Notifications and Records. For example, the following information can be pushed: (1) Root cause: SSD hardware failure; (2) Topology location: Cluster C1 / Replication group R2 / Follower N1; (3) Execution actions: Node removal, instance migration; (4) Execution result: Success.

[0119] Example 2: Table Access Availability Decision Maker 1. Scenario: The availability of read / write operations on storage tables decreases, for example, failing to meet set targets such as a 99.99% read / write success rate. For instance, if the network card of a machine hosting a storage node fails, continuous access timeouts occur when the application tries to read / write tables via the SDK. However, the storage node itself is unaware that the application SDK initiated and failed requests. The table data is distributed across dozens to thousands of storage nodes, requiring accurate and rapid identification of whether the availability degradation is caused by the failure of a small number of machines.

[0120] 2. Problem Detection Process: When the business SDK reads or writes tables, it will automatically report to the SDK access status reporting center at regular intervals, including the number of read / write operations, the number of failures (such as flow control and timeout failures), the SDK business instance, and the storage node machine and service port where the access failed. When the SDK access status reporting center detects that the availability of the business access table cannot meet the preset value, such as 99.99%, it will start to call back to notify the self-healing system to start analyzing the fault.

[0121] 3. The overall architecture is as follows: (1) Dual-perspective verification: accurate identification of high failure rate of the server + backup recall of the server general decision-maker, covering two scenarios: "clear failure" and "suspected failure".

[0122] (2) Three-level downgrade analysis: precise analysis, fallback recall, SDK-assisted notification, gradually relax the conditions but simultaneously reduce the intensity of operation.

[0123] (3) Dual-window design: The SDK access status reporting center acts as the statistical proxy layer on the SDK (client) access side. It performs rolling aggregation of table-level access metrics reported by business clients according to a preset period (e.g., 3 minutes), forming a statistical slice for each period. The system maintains the data of the two most recent consecutive slices in memory. For example: -slice1: data of the previous complete window, representing the recent stable state; -slice2: data of the current or latest completed window, reflecting the recent trend.

[0124] 4. The decision-making process of the decision-maker is as follows: Level 1, Precise Analysis: This involves traversing the API interfaces of each instance in the SDK access status reporting center to obtain the two consecutive shard data maintained on each instance. Global normalized summation is then performed: the total number of requests is summed, and the counts of various read / write timeouts / failures are summed separately. Simultaneously, the proportion of failed storage nodes to the total number of requests is aggregated. If the failure rate of a certain storage node exceeds a preset value (e.g., 80%), the result is considered directly reliable. Self-healing operations at the execution layer can then proceed.

[0125] The second level, fallback recall: Select all storage nodes whose access failure rate exceeds a preset value (e.g., 5%), and call the general decision-maker for secondary fault confirmation to analyze whether the nodes have anomalies such as hardware failures or machine / environmental failures. If the secondary confirmation of a fault occurs, the self-healing execution layer can continue with self-healing operations.

[0126] Level 3, SDK-assisted analysis: If neither precise analysis nor fallback recall reveals any storage node anomalies, the access timeout may be due to a fault in the SDK service instance itself. First, proactively retrieve the runtime logs of the SDK access status reporting center instance, filter and aggregate the SDK service instances and their respective machines. Analyze whether there is clustering of SDK service instances; if a particular service instance accounts for more than a preset value of all failures (e.g., 60%), then it is highly likely that the problem is caused by the SDK service instance. No self-healing processing is needed on the storage side; simply automatically send the analysis and decision-making results to the upper-level decision-making end, such as the decision-making equipment of on-duty personnel.

[0127] Once a decision is made in any of the above steps, the system can enter the self-healing execution layer to prepare for the execution of the self-healing strategy. If the first-level precise analysis or the second-level fallback recall accurately identifies an abnormal storage node, the system can enter the self-healing execution layer.

[0128] 5. Multi-layered security mechanisms can be embedded during the execution of the obtained self-healing strategy, specifically including: (1) System-level start-stop control: The decision-making equipment of the upper-level decision-making end, such as the decision-making equipment of the duty personnel, can temporarily close the overall automatic self-healing execution permission through the self-healing start-stop intervention interface (such as restricting changes when there is a large-scale host environment failure).

[0129] (2) Cluster-level start-up and shutdown control: The upper-level decision-making end can temporarily disable the automatic self-healing execution permission of a certain cluster through the self-healing cluster start-up and shutdown intervention interface (such as restricting changes during the observation of a new function going online in a certain cluster).

[0130] (3) Self-healing frequency control, supporting each cluster to independently configure the maximum number of healing operations per day, such as the number of automatic primary switching, the number of automatic replica removal (for example, first remove abnormal replicas, the number of replicas is reduced from 3 to 2, and then a new storage node is pulled in and the failure is recovered to 3 replicas), and the number of automatic instance migration (for example, if the machine where the current storage node is located has non-continuous failures, a new instance is first transferred from PaaS, the number of replicas is increased from 3 to 4, and then abnormal replicas are removed and the instance is deleted and returned to PaaS).

[0131] (4) Over-frequency degradation notification: When the number of automatic self-healing times per day of the cluster exceeds the threshold, it supports automatically sending self-healing commands and fault analysis decision results to the upper-level decision-making end, and manual judgment on whether to execute.

[0132] (5) Replication group availability protection, such as a 3-replica configuration (one master and two slaves). During self-healing, if the corresponding replica of a storage node needs to be automatically removed, the latest topology status information of the storage node will be obtained through the metadata management service, and the current replica role will be automatically verified. If the master role is triggered first, an election will be automatically triggered to switch the master role to a slave role. If the slave role is activated, it is necessary to verify whether there are currently 3 replicas and all of them are available. Otherwise, replica removal cannot continue (for example, it is allowed for 3 replicas to become 2 replicas first, and then restore to 3 replicas. However, it is not allowed for 2 replicas to continue removing replicas to become 1 replica, leaving only one master role and no slave roles). A storage node in a replication group may be either a master role or a slave role. PaaS may include enterprise content container orchestration services.

[0133] Example 3: Master-Slave Synchronization Gap Decision Maker 1. Target Scenario: This scenario targets a storage cluster where the engine's underlying layer is a proactive, automatic data synchronization mechanism. This means that after business data is written to the master node via the SDK, it is automatically synchronized to all slave nodes. Each data write generates a monotonically increasing sequence number. Under normal circumstances, the data between the master and slave nodes should be synchronized quickly and promptly. However, when a slave node experiences hardware failure, network jitter, or disk performance degradation, causing log synchronization to lag (in some scenarios, such as high write traffic but low read traffic for the business SDK, the table access availability decision-maker may not detect the anomaly), the difference between the latest data sequence numbers of the master and slave nodes grows increasingly large. This can lead to the business SDK being unable to retrieve the latest written data. Each cluster has thousands to tens of thousands of replication groups. It is necessary to be aware of the replication group to which a storage node belongs, which replicas it contains, and the latest data sequence number of each replica. It is also necessary to accurately identify shards with significant differences between master and slave data, and quickly identify and remove faulty nodes, restoring them to normal machines to avoid timeliness issues for businesses accessing old data.

[0134] 2. Problem Awareness Process: Storage nodes periodically interact with the cluster management service via heartbeat requests, recording the latest data sequence number in the current replica within the heartbeat information. Upon receiving the sequence numbers of all master and slave nodes in the same replication group, the cluster management service compares the data sequence numbers of the slave nodes with those of the master nodes. Once the slave node's sequence number lags behind the master node by a preset value (e.g., 1000), it initiates a callback to notify the self-healing system to begin fault analysis.

[0135] 3. The overall design architecture of the master-slave synchronization gap decision-maker can adopt a fault decision-making method based on reverse tracing of cluster management service logs and secondary verification of node health status.

[0136] 4. The decision-making process of the decision-maker is as follows: (1) Analyze the cluster management service instance that issued the alarm, actively pull the latest logs of the instance, and filter out logs related to excessive data differences. Parse the cluster name, replication group number, master-slave replica information, and log difference value from the logs. Obtain all master-slave storage nodes in the corresponding replication group through the metadata management service, and actively request the storage node to query the latest sequence number interface. Determine the latest data sequence number to find the slave nodes whose difference from the master node reaches the threshold.

[0137] (2) For slave nodes that are significantly different from the master node, the general decision-maker is invoked to perform secondary fault confirmation and analyze whether the node has any anomalies such as hardware failure or machine environment failure. If the secondary confirmation of fault shows that only one slave node is faulty, the self-healing execution layer self-healing operation can continue. If more than one slave node is faulty, only a fault identification notification is sent to the upper-level decision-making end, and no self-healing operation is actively performed.

[0138] Example 4: Instance Exception Decision Maker 1. Target Scenarios: Abnormal scenarios perceived by the storage node instance itself, such as excessively long read / write engine latency within the storage node, backlog of external request processing, and process startup failures. (This is complementary to the table access availability decision-maker; one detects access node failures from the outside, while the other detects abnormal request processing metrics from within the node). In a distributed database cluster, storage nodes serve as the basic units for data carrying and computation execution, and their operational status directly affects the read / write availability of upper-layer services. However, node failures manifest in various ways: they can be at the operating system level (SSH unreachability, monitoring failure), at the hardware level (such as SSD lifespan expiration, hardware errors), or at the resource contention level (CPU overload or network quality degradation).

[0139] Traditional threshold alarms based on a single indicator struggle to distinguish between "severe single points requiring immediate migration / master switching" and "sub-healthy states that can continue to be observed," and are prone to unnecessary self-healing operations due to misidentification. Therefore, a multi-dimensional instance-level anomaly comprehensive decision-making mechanism can complete root cause aggregation and repair path selection without relying on external manual judgment.

[0140] 2. Problem detection process: Set callback rules in the alarm system. When the specified operating indicators of the storage node instance reach the threshold, a callback is sent to notify the self-healing system to start analyzing the fault.

[0141] 3. The overall design architecture of the instance anomaly decision-maker can adopt a hierarchical short-circuit detection method, which sequentially calls multiple heterogeneous detection capabilities in a specific order. A specific example is as follows: (1) Network connectivity verification; (2) Low-level hardware health check; (3) Runtime resource pressure assessment and network performance degradation detection.

[0142] Each test can be set with an independent decision threshold, and subsequent tests will be terminated immediately after any test is hit in order to quickly converge the results.

[0143] 4. The decision-making process of the decision-maker is as follows: (1) For storage node processes that fail to start, actively pull the corresponding instance exception logs. Analyze the root causes of the startup failure in the logs, such as port being occupied, SSD sector size not meeting the storage engine's expectations, or failure to interact with the PaaS Agent. For such basic environmental issues, promptly send the analysis results to the upper-level decision-making end, such as the decision-making equipment of the on-duty personnel, and simultaneously initiate instance migration for the storage node, allowing PaaS to reallocate instances on other clusters.

[0144] (2) For abnormalities where the engine reads and writes data for too long, the general decision-maker is called to analyze the hardware level first (such as the expiration of SSD erase and write life, hardware error, machine CPU load). If the fault is confirmed, the self-healing execution layer self-healing operation can continue.

[0145] (3) For external request processing backlog exceptions, call the general decision-maker to analyze the fault. If the fault is confirmed, the self-healing execution layer self-healing operation can continue.

[0146] According to the embodiments of this disclosure, a full-link scenario-adaptive self-healing or / and self-healing operation decision-making process can be realized, which can improve the accuracy and effectiveness of the decision results.

[0147] According to the embodiments of this disclosure, this solution can achieve one or more of the following technical effects: 1. To address the issue of "single perception dimension", multi-dimensional collaborative perception has been achieved, covering the entire scenario. It not only focuses on node indicators, but also on the perception of the entire link, such as business access, cluster and replication group status, and topology consistency.

[0148] 2. To address the issue of "polling-based perception lag," an event-driven fault detection mechanism has been implemented, which can detect anomalies in seconds. Compared to polling, the perception latency is significantly reduced. The monitoring system is reused, eliminating the need for additional computing resources to detect anomalies.

[0149] 3. To address the issue of "weak root cause analysis capabilities," the system has achieved root cause analysis capabilities through joint attribution by a dedicated decision-maker and a general health checker. By fusing and analyzing multi-source heterogeneous data, the false positive rate can be significantly reduced.

[0150] 4. To address the issue of "treating stateful services as stateless", we have implemented fault analysis and self-healing capabilities for stateful distributed storage. This enables us to identify master-slave roles, perceive replica health status, perceive topology location and characteristics, differentiate expansion suggestions, remove replicas, switch master and slave, migrate instances, and perform manual intervention. This solves the secondary fault problems caused by treating stateful storage services as stateless services.

[0151] 5. To address the "lack of a safety fallback mechanism," measures such as replica count protection, frequency control, global / local start / stop, and excessively frequent degradation have been implemented to achieve controlled self-healing behavior.

[0152] 6. To address the issue of "unobservable self-healing process", the system supports post-hoc verification and continuous optimization by recording complete logs and pushing results, making the entire self-healing process transparent and traceable.

[0153] The embodiments disclosed herein also achieve risk prevention by early detection, prevention of deterioration, and mitigation of minor risks into major ones; breaking the alarm vicious cycle, and realizing multi-dimensional fault perception through an overall architecture of "multi-dimensional perception, precise decision-making, refined self-healing, safety net, and observable closed-loop self-healing".

[0154] Figure 9 This is a schematic diagram of the structure of a self-healing device 900 for a stateful data system according to an embodiment of the present disclosure. In one embodiment, the device may include: Receiver module 910 is used to receive anomaly detection information from a stateful data system; The acquisition module 920 is used to acquire the stateful context in the stateful data system based on the anomaly perception information. The determination module 930 is used to determine the current scene based on the anomaly perception information; The module 940 is used to invoke the scene decision-maker corresponding to the current scene and obtain the scene decision result based on the stateful context. Selection module 950 is used to dynamically select a self-healing strategy when an abnormal node is determined based on the scenario decision results.

[0155] In one implementation, the current scenario is a disk space scenario, and the disk space scenario is determined by: determining that the disk space is insufficient based on anomaly perception information from the stateful data system; The process of invoking the scene decision-maker corresponding to the current scene and obtaining the scene decision result based on the stateful context includes: Call the disk space decision-maker corresponding to the disk space scenario to query the topological location of the alarm node in the stateful context; Based on the topological location of the alarm node, a horizontal comparison of replicas is performed to obtain the horizontal comparison result.

[0156] In one implementation, performing a replica lateral comparison based on the topological location of the alarm node includes: Based on the topological location of the alarm node, the replication group to which the abnormal node belongs and all its replica nodes are identified; Analyze the disk usage status of all replica nodes within the replication group; If the disk usage of multiple replica nodes exceeds the threshold, it is determined that the overall write volume is too large, and a capacity expansion suggestion is output. If only the replica node represented by the alarm node exceeds the threshold, it is determined to be a single point of failure.

[0157] In one implementation, the current scenario is an access availability scenario, and the method for determining the access availability scenario includes: determining, based on the anomaly perception information from the access status reporting center, that the availability of the business access table in the data node does not meet a set threshold. The process of invoking the scene decision-maker corresponding to the current scene and obtaining the scene decision result based on the stateful context includes: Invoke the availability decision-maker corresponding to the availability scenario, traverse the interfaces of each node instance of the business access status reporting center in the stateful context, and obtain the continuous sharded data maintained on each node instance. The access requests for the continuously segmented data are statistically analyzed to obtain global statistical results; If, in the global statistics, the access failure rate of one or more data nodes exceeds the first access threshold, the one or more data nodes will be identified as abnormal nodes. If there are no nodes in the global statistics whose access failure rate exceeds the first access threshold, then abnormal nodes whose access failure rate exceeds the second access threshold are filtered out. The second access threshold is less than the first access threshold.

[0158] In one implementation, the current scenario is a master-slave synchronization gap scenario, and the method for determining the master-slave synchronization gap scenario includes: determining that the master-slave synchronization gap is greater than a set threshold based on anomaly perception information from the cluster management service. The process of invoking the scene decision-maker corresponding to the current scene and obtaining the scene decision result based on the stateful context includes: Invoke the master-slave synchronization gap decision-maker corresponding to the master-slave synchronization gap scenario, and obtain all master-slave nodes of the replication group whose master-slave synchronization gap is greater than a set threshold through the stateful context from the meta information management service; Based on the latest sequence number query interface of the master-slave node, identify slave nodes whose difference from the master node reaches a threshold. These slave nodes are considered abnormal nodes.

[0159] In one implementation, the current scenario is an instance anomaly scenario, and the method for determining the instance anomaly scenario includes: determining that the data node is anomaly based on the anomaly perception information from the data node of the stateful data system; The process of invoking the scene decision-maker corresponding to the current scene and obtaining the scene decision result based on the stateful context includes: Invoke the instance exception decision maker corresponding to the instance exception scenario. For exceptions where a node cannot start, actively pull the instance exception log of the node that cannot start. Analyze the root cause of the startup failure in the aforementioned exception log; For the underlying environmental issues identified in the root cause, an analysis conclusion is output, and an instance task to migrate the data node is initiated to redistribute the data node's data on other clusters.

[0160] Figure 10 This is a schematic diagram of the structure of a self-healing device 1000 for a stateful data system according to an embodiment of this disclosure. The device may include: a receiving module 1010, an acquiring module 1020, a determining module 1030, a calling module 1040, and a selecting module 1050. The functions and roles of each module can be found in the description of the self-healing device 900 for the stateful data system described above, and will not be repeated here. In one embodiment, before dynamically selecting a self-healing strategy, the following steps are included: The inspection module 1060 is used to call the general decision-maker to perform a health check on the abnormal node, and to reconfirm whether the abnormal node is abnormal based on the health check results, and / or the root cause of the abnormality in the current scenario.

[0161] In one implementation, such as Figure 10 As shown, before dynamically selecting the self-healing strategy, a verification module 1070 is included, which performs one or more of the following security checks: Check whether the number of available replicas in the replication group meets the threshold; replica nodes below the threshold are not allowed to be removed. Based on historical self-healing records, check whether the replica node represented by the abnormal node has been subjected to self-healing operations multiple times within a set time period in the past. If the frequency of self-healing operations exceeds the upper limit, replace the self-healing operation with a notification. Check if the system-level self-healing switch is enabled; if not, disable the global self-healing capability. Check if the self-healing switch of the current cluster is enabled; if not, disable the self-healing capability of the current cluster.

[0162] In one implementation, the self-healing strategy is dynamically selected, including one or more of the following: If the abnormal node is the master node, a master-slave switch will be performed first. If the abnormal node is a slave node and the health status of the replica node represented by the abnormal node meets the availability requirements, the abnormal node is removed, and a new node is started to perform abnormal recovery of the new replica node. If the abnormal node is determined to be a hardware malfunction based on the health check results of the abnormal node, the instance of the abnormal node shall be expelled or deleted. If the disk usage of multiple replica nodes in a replication group exceeds the threshold, it is determined that the overall write volume is too large, and a capacity expansion suggestion is output. If the number of available replicas in the replication group is insufficient, an exception notification is output but no self-healing operation is performed; the fault is waited for recovery. If the environment is normal but the service is abnormal, suggest manual intervention.

[0163] In one implementation, such as Figure 10 As shown, the device may also include: an operation and maintenance module 1080, used to record and / or push one or more of the following information: the root cause of the current scene anomaly, the topological location of the abnormal node, the self-healing operation performed on the abnormal node, and the self-healing result after the self-healing operation is performed.

[0164] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0165] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0166] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0167] Figure 11 A schematic block diagram of an example electronic device 1100 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0168] like Figure 11 As shown, device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1102 or a computer program loaded into random access memory (RAM) 1103 from storage unit 1108. The RAM 1103 may also store various programs and data required for the operation of device 1100. The computing unit 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. Input / output (I / O) interface 1105 is also connected to bus 1104.

[0169] Multiple components in device 1100 are connected to I / O interface 1105, including: input unit 1106, such as keyboard, mouse, etc.; output unit 1107, such as various types of monitors, speakers, etc.; storage unit 1108, such as disk, optical disk, etc.; and communication unit 1109, such as network card, modem, wireless transceiver, etc. Communication unit 1109 allows device 1100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0170] The computing unit 1101 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above, such as the self-healing method for a stateful data system. For example, in some embodiments, the self-healing method for a stateful data system may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1100 via ROM 1102 and / or communication unit 1109. When the computer program is loaded into RAM 1103 and executed by the computing unit 1101, one or more steps of the self-healing method for a stateful data system described above may be performed. Alternatively, in other embodiments, computing unit 1101 may be configured by any other suitable means (e.g., by means of firmware) to perform a self-healing method for a stateful data system.

[0171] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0172] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0173] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0174] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0175] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0176] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0177] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0178] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A self-healing method for a stateful data system, comprising: Receive anomaly detection information from stateful data systems; Based on the anomaly detection information, obtain the stateful context in the stateful data system; Based on the anomaly detection information, the current scene is determined; Invoke the scene decision-maker corresponding to the current scene, and obtain the scene decision result based on the stateful context; If an abnormal node is identified based on the scenario decision results, a self-healing strategy is dynamically selected.

2. The method according to claim 1, wherein, The current scenario is a disk space scenario, and the disk space scenario is determined by: determining that there is insufficient disk space based on the anomaly perception information from the stateful data system; The process of invoking the scene decision-maker corresponding to the current scene and obtaining the scene decision result based on the stateful context includes: Call the disk space decision-maker corresponding to the disk space scenario to query the topological location of the alarm node in the stateful context; Based on the topological location of the alarm node, a horizontal comparison of replicas is performed to obtain the horizontal comparison result.

3. The method according to claim 2, wherein, Based on the topological location of the alarm node, a replica lateral comparison is performed, including: Based on the topological location of the alarm node, the replication group to which the abnormal node belongs and all its replica nodes are identified; Analyze the disk usage status of all replica nodes within the replication group; If the disk usage of multiple replica nodes exceeds the threshold, it is determined that the overall write volume is too large, and a capacity expansion suggestion is output. If only the replica node represented by the alarm node exceeds the threshold, it is determined to be a single point of failure.

4. The method according to claim 1, wherein, The current scenario is an access availability scenario. The method for determining the access availability scenario includes: determining, based on the anomaly perception information from the access status reporting center, that the availability of the business access table in the data node does not meet the set threshold. The process of invoking the scene decision-maker corresponding to the current scene and obtaining the scene decision result based on the stateful context includes: Invoke the availability decision-maker corresponding to the availability scenario, traverse the interfaces of each node in the business access status reporting center in the stateful context, and obtain the continuous sharded data maintained on each node. The access requests for the continuously segmented data are statistically analyzed to obtain global statistical results; If, in the global statistics, the access failure rate of one or more data nodes exceeds the first access threshold, the one or more data nodes will be identified as abnormal nodes. If there are no nodes in the global statistics whose access failure rate exceeds the first access threshold, then abnormal nodes whose access failure rate exceeds the second access threshold are filtered out. The second access threshold is less than the first access threshold.

5. The method according to claim 1, wherein, The current scenario is a master-slave synchronization gap scenario, and the method for determining the master-slave synchronization gap scenario includes: determining that the master-slave synchronization gap is greater than a set threshold based on the anomaly perception information from the cluster management service; The process of invoking the scene decision-maker corresponding to the current scene and obtaining the scene decision result based on the stateful context includes: Invoke the master-slave synchronization gap decision-maker corresponding to the master-slave synchronization gap scenario, and obtain all master-slave nodes of the replication group whose master-slave synchronization gap is greater than a set threshold through the stateful context from the meta information management service; Based on the latest sequence number query interface of the master-slave node, identify slave nodes whose difference from the master node reaches a threshold. These slave nodes are considered abnormal nodes.

6. The method according to claim 1, wherein, The current scenario is an instance anomaly scenario, and the method for determining the instance anomaly scenario includes: determining that the data node is anomaly based on the anomaly perception information from the data node of the stateful data system; The process of invoking the scene decision-maker corresponding to the current scene and obtaining the scene decision result based on the stateful context includes: Invoke the instance exception decision maker corresponding to the instance exception scenario. For exceptions where a node cannot start, actively pull the instance exception log of the node that cannot start. Analyze the root cause of the startup failure in the aforementioned exception log; For the underlying environmental issues identified in the root cause, an analysis conclusion is output, and an instance task to migrate the data node is initiated to redistribute the data node's data on other clusters.

7. The method according to any one of claims 1 to 6, prior to dynamically selecting a self-healing strategy, includes: The general decision-maker is invoked to perform a health check on the abnormal node. Based on the health check results, it is confirmed again whether the abnormal node is abnormal and / or the root cause of the abnormality in the current scenario.

8. The method according to any one of claims 1 to 7, wherein, Before dynamically selecting a self-healing strategy, one or more of the following security checks must be performed: Check if the number of available replicas in the replication group meets the threshold; replica nodes below the threshold are not allowed to be removed. Based on historical self-healing records, check whether the replica node represented by the abnormal node has been subjected to self-healing operations multiple times within a set time period in the past. If the frequency of self-healing operations exceeds the upper limit, replace the self-healing operation with a notification. Check if the system-level self-healing switch is enabled; if not, disable the global self-healing capability. Check if the self-healing switch of the current cluster is enabled; if not, disable the self-healing capability of the current cluster.

9. The method according to any one of claims 1 to 8, wherein, Dynamically select self-healing strategies, including one or more of the following: If the abnormal node is the master node, a master-slave switch will be performed first. If the abnormal node is a slave node and the health status of the replica node represented by the abnormal node meets the availability requirements, the abnormal node is removed, and a new node is started to perform abnormal recovery of the new replica node. If the abnormal node is determined to be a hardware malfunction based on the health check results of the abnormal node, the instance of the abnormal node shall be expelled or deleted. If the disk usage of multiple replica nodes in a replication group exceeds the threshold, it is determined that the overall write volume is too large, and a capacity expansion suggestion is output. If the number of available replicas in the replication group is insufficient, an exception notification is output but no self-healing operation is performed; the fault is waited for recovery. If the environment is normal but the service is abnormal, suggest manual intervention.

10. The method according to any one of claims 1 to 9, further comprising: Record and / or push one or more of the following information: the root cause of the anomaly in the current scenario, the topological location of the abnormal node, the self-healing operation performed on the abnormal node, and the self-healing result after the self-healing operation is performed.

11. A self-healing device for a stateful data system, comprising: The receiving module is used to receive anomaly detection information from stateful data systems; The acquisition module is used to acquire the stateful context in the stateful data system based on the anomaly perception information. The determination module is used to determine the current scene based on the anomaly perception information; The invocation module is used to invoke the scene decision-maker corresponding to the current scene and obtain the scene decision result based on the stateful context. The selection module is used to dynamically select a self-healing strategy when an abnormal node is identified based on the scenario decision results.

12. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-10.

13. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-10.

14. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-10.

Images