Fault processing method and device of database, computer equipment and readable storage medium

Abstract

The invention relates to a database fault processing method and device, computer equipment and a readable storage medium, and is applied to a management and control center. The method comprises the following steps: storing node information of each database instance in a target resource pool; the node information comprises an IP address and a port; based on the node information, initiating fault detection and operation state monitoring to nodes of the database instances in the target resource pool to obtain fault detection results and operation state information of the database instances; analyzing and calculating the fault detection result and the operation state information of each database instance to obtain an analysis and calculation result; judging whether the database instance has a preset fault type or not based on the analysis and calculation result; and if the preset fault type exists, executing a self-recovery task corresponding to the preset fault type. By adopting the method, each database instance can autonomously detect and judge the fault type and execute the corresponding fault self-recovery task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cloud database monitoring, in particular to a database fault processing method and device, computer equipment and readable storage medium. BACKGROUND

[0002] MongoDB is a document-based NoSQL database, which is becoming more and more popular due to its flexible data model, excellent read-write ability and other characteristics. In a public cloud architecture, cloud service providers often need to manage a large number of MongoDB instances.

[0003] However, these MongoDB instances may have unstable performance or even abnormal conditions when facing high load or improper user operations, so the running state of MongoDB directly affects the progress of user system business. Therefore, it is particularly important to monitor MongoDB in real time, discover MongoDB abnormalities in time and issue warnings, and perform self-repair in the case of MongoDB abnormalities to ensure stable operation of the database.

[0004] Therefore, how to process the fault of MongoDB (database) has become a problem to be solved. SUMMARY

[0005] Therefore, it is necessary to provide a database fault processing method, device, computer equipment and readable storage medium capable of monitoring, analyzing and recovering MongoDB faults in view of the above technical problems.

[0006] In a first aspect, the present application provides a database fault processing method applied to a management and control center, comprising:

[0007] storing node information of each database instance in a target resource pool; the node information includes IP address and port;

[0008] Based on the node information, initiating fault detection and running state monitoring to the nodes of each database instance in the target resource pool to obtain fault detection results and running state information of each database instance;

[0009] analyzing and calculating the fault detection results and running state information of each database instance to obtain analysis and calculation results;

[0010] Based on the analysis and calculation results, determining whether the database instance has a preset fault type;

[0011] If the preset fault type exists, a self-recovery task corresponding to the preset fault type is executed.

[0012] In one of the embodiments, based on the node information, the fault detection and running state monitoring are initiated to the nodes of each database instance in the target resource pool to obtain the fault detection result and the running state information of each database instance, including:

[0013] Based on the IP address and the port in the node information, the network connectivity test and the heartbeat detection are performed on the nodes of each database instance to confirm whether the nodes are online;

[0014] The connection is established to the nodes detected as online and the state query command is executed to obtain the process survival state and the running state data of the nodes;

[0015] The results of the network connectivity test failure, the heartbeat detection non-response, and the state query command execution exception are included in the fault detection result;

[0016] The online state, the process survival state, and the detailed running state data of the nodes of each database instance are included in the running state information.

[0017] In one of the embodiments, the preset fault type includes instance fault; based on the analysis calculation result, it is determined whether the database instance has the preset fault type, including:

[0018] When the deployment architecture of the database instance is a sharding cluster, based on the analysis calculation result, the state data of any one of the routing node, the configuration node, and each shard data node of the database instance is counted;

[0019] If the state data meets the target condition, it is determined that the database instance has instance fault;

[0020] The state data meeting the target condition includes any one of the following:

[0021] All routing nodes are offline or unavailable;

[0022] The configuration node has no master node;

[0023] Any shard data node has no master node.

[0024] In one of the embodiments, the preset fault type includes slow query overload; based on the analysis calculation result, it is determined whether the database instance has the preset fault type, including:

[0025] Based on the analysis calculation result, the database system configuration file query command is executed to extract the slow query log of each database instance;

[0026] The extracted slow query log is classified by hash according to the query statement characteristics, and the occurrence frequency of each type of slow query is counted;

[0027] If the occurrence frequency of the target class slow query exceeds the preset frequency threshold, and the occurrence frequency of the target class slow query is confirmed to be gradually increasing through time series analysis, it is determined that the database instance corresponding to the target class slow query has slow query overload.

[0028] In one of the embodiments, the preset fault type includes excessive master-slave delay; based on the analysis calculation result, it is determined whether the database instance has the preset fault type, including:

[0029] When the deployment architecture of the database instance is a replica set, based on the analysis calculation result, the operation timestamp information of the master node and the slave node is obtained by executing a replica set state query command, and the master-slave delay is calculated;

[0030] In the case where the master-slave delay exceeds the preset time threshold, and the master-slave delay is confirmed to be gradually increasing through time series analysis, a replica set replication information query command is executed to query the window size of the operation log of the replica set;

[0031] If the master-slave delay has exceeded the window size of the operation log, it is determined that the database instance has the excessive master-slave delay fault.

[0032] In one of the embodiments, the self-recovery task corresponding to the preset fault type is executed, including:

[0033] If the preset fault type is node fault, and the deployment architecture of the database instance is single node or replica set, the self-recovery task executed includes: loading the context information of the fault node, calling the database node start command, and waiting for the node process to complete initialization;

[0034] If the preset fault type is instance fault, and the deployment architecture of the database instance is shard cluster, the self-recovery task executed includes: sorting the fault nodes in the order of configuration node, shard data node, and routing node, and executing the corresponding recovery subtask flow for each type of fault node;

[0035] If the preset fault type is excessive master-slave delay, and the deployment architecture of the database instance is replica set or data node replica set in shard cluster, the self-recovery task executed includes: obtaining the latest valid hot backup data of the database instance; copying data from the master node or the healthy backup node to the fault slave node; performing initialization operation on the fault slave node, and rejoining the initialized slave node to the replica set;

[0036] If the preset fault type is slow query overload, the self-recovery task executed includes: retrieving the database table and field corresponding to the slow query overload; analyzing whether there is no matching index established in the database table and field; generating a recommended strategy for establishing the target index and issuing a warning; synchronizing the slow query analysis result and the recommended strategy to the log system.

[0037] Secondly, this application also provides a database fault handling device, applied in a control center, comprising:

[0038] The storage module is used to store node information for each database instance within the target resource pool. The node information includes IP address and port.

[0039] The detection module is used to initiate fault detection and operation status monitoring to the nodes of each database instance in the target resource pool based on node information, so as to obtain the fault detection results and operation status information of each database instance.

[0040] The analysis and calculation module is used to analyze and calculate the fault detection results and running status information of each database instance to obtain the analysis and calculation results;

[0041] The fault diagnosis module is used to determine whether a database instance has a preset fault type based on the analysis and calculation results.

[0042] The self-recovery module is used to assemble a task flow corresponding to a preset fault type if a preset fault type exists, and to execute a fault self-recovery process based on the task flow.

[0043] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0044] Stores node information for each database instance within the target resource pool; node information includes IP address and port.

[0045] Based on node information, fault detection and operation status monitoring are initiated on the nodes of each database instance in the target resource pool to obtain the fault detection results and operation status information of each database instance.

[0046] The fault detection results and running status information of each database instance are analyzed and calculated to obtain the analysis and calculation results;

[0047] Based on the analysis and calculation results, determine whether the database instance has a preset fault type;

[0048] If a preset fault type exists, the self-recovery task corresponding to the preset fault type will be executed.

[0049] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0050] Stores node information for each database instance within the target resource pool; node information includes IP address and port.

[0051] Based on the node information, initiate fault detection and running state monitoring to the nodes of each database instance in the target resource pool to obtain the fault detection result and running state information of each database instance;

[0052] Analyze and calculate the fault detection result and running state information of each database instance to obtain an analysis and calculation result;

[0053] Based on the analysis and calculation result, determine whether the database instance has a preset fault type;

[0054] If the preset fault type exists, execute a self-recovery task corresponding to the preset fault type.

[0055] In a fifth aspect, the present application also provides a computer program product, comprising a computer program which, when executed by a processor, implements the following steps:

[0056] Store node information of each database instance in a target resource pool; the node information includes an IP address and a port;

[0057] Based on the node information, initiate fault detection and running state monitoring to the nodes of each database instance in the target resource pool to obtain the fault detection result and running state information of each database instance;

[0058] Analyze and calculate the fault detection result and running state information of each database instance to obtain an analysis and calculation result;

[0059] Based on the analysis and calculation result, determine whether the database instance has a preset fault type;

[0060] If the preset fault type exists, execute a self-recovery task corresponding to the preset fault type.

[0061] The above database fault processing method, device, computer equipment and readable storage medium, in the scenario of needing to manage a large number of MongoDB instances, based on the node information of each database instance, initiate fault detection to the nodes of each database instance, for the running state with a trend of failure, perform data calculation and analysis, so that each database instance can autonomously detect and determine the fault type, and execute the corresponding fault self-recovery task, which can not only reduce the mistakes and delay of manual intervention operation, but also reduce the probability of business system failure and shorten the recovery time. BRIEF DESCRIPTION OF DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application or the related art. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other related drawings can also be obtained on the basis of these drawings without any creative effort.

[0063] Figure 1 An application environment diagram of the database fault processing method in one embodiment;

[0064] Figure 2 A flowchart of the database fault processing method in one embodiment;

[0065] Figure 3 A flowchart of the database fault processing method in another embodiment;

[0066] Figure 4 A flowchart of the database fault processing method in another embodiment;

[0067] Figure 5 A flowchart of the database fault processing method in another embodiment;

[0068] Figure 6 A flowchart of the database fault processing method in another embodiment;

[0069] Figure 7 A flowchart of the database fault processing method in another embodiment;

[0070] Figure 8 A flowchart of the database fault processing method in another embodiment;

[0071] Figure 9 A flowchart of the database fault processing method in another embodiment;

[0072] Figure 10 A structural block diagram of the database fault processing apparatus in one embodiment. DETAILED DESCRIPTION

[0073] In order to make the purposes, technical solutions and advantages of the present application more clear, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0074] MongoDB is a document-based NoSQL database that is becoming increasingly popular due to its flexible data model, excellent read-write capabilities, and other features. In a public cloud architecture, cloud service providers often need to manage a large number of MongoDB instances.

[0075] However, these MongoDB instances may experience unstable performance or even abnormal situations under high load or improper user operations, and thus the running state of MongoDB directly affects the progress of user system business. Therefore, it is particularly important to monitor MongoDB in real time, timely detect MongoDB abnormalities and issue warnings, and perform self-repair in the case of MongoDB abnormalities to ensure stable operation of the database. Therefore, how to handle the failure of MongoDB (database) has become a problem to be solved. Based on this, the present application provides a database failure handling method to solve this problem.

[0076] In the following, the application environment of the database failure handling method provided by the present application will be described. The database failure handling method provided by the embodiments of the present application can be applied to the management and control center as shown in Figure 1 The management and control center can be a server. The management and control center includes a processor, a memory, an input / output interface (I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the management and control center is used to provide computing and control capabilities. The memory of the management and control center includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the management and control center is used to store related data of the database instance. The input / output interface of the management and control center is used to exchange information between the processor and external devices. The communication interface of the management and control center is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement a database failure handling method.

[0077] Those skilled in the art can understand that Figure 1 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the management and control center to which the scheme of the present application is applied. The specific management and control center can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0078] In an exemplary embodiment, as shown in Figure 2 A database failure handling method is provided, and the method is applied toFigure 1 The management center in the target resource pool is taken as an example for illustration, including:

[0079] S201, store node information of each database instance in the target resource pool; the node information includes IP address and port.

[0080] The management center can include multiple resource pools, each resource pool can store node information of at least one database instance, and the target resource pool refers to one of the multiple resource pools of the management center.

[0081] The database instance can be a MongoDB instance, and the node information of the database instance includes the IP address of the database instance and the port of the database instance.

[0082] In this embodiment, the management center can pre-store the IP address and port of each database instance in the target resource pool. In addition, after the management center pre-stores the IP address and port of each database instance in the target resource pool, the firewall whitelist of each database instance in the target resource pool can be released, so that the management center can initiate fault detection behavior to multiple database instances in the target resource pool.

[0083] S202, based on the node information, initiate fault detection and running state monitoring to the nodes of each database instance in the target resource pool, to obtain fault detection results and running state information of each database instance.

[0084] The fault detection is used to detect whether the node of the database instance fails, and the fault detection result includes normal detection result and fault detection result, etc. The fault detection result includes node failure, instance failure, slow query overload, and large standby delay, etc.

[0085] The running state monitoring is used to monitor the running state of the node of the database instance, to obtain the running state information of the node of the database instance. The running state information includes health score, real-time load, availability level, and predictive failure trend, etc.

[0086] In this embodiment, after the management center stores the node information of each database instance in the target resource pool, it can initiate fault detection and running state monitoring to the nodes of each database instance in the target resource pool based on the node information, to obtain fault detection results and running state information of each database instance.

[0087] Optionally, the IP addresses and ports of all database instance nodes can be acquired first, and then the monitoring engine of the management center initiates a multi-thread monitoring task to each database instance in the target resource pool in parallel, establishes a TCP / short connection to each node corresponding to the database instance, sends a heartbeat probe to each node corresponding to the database instance within a timeout threshold, and collects the returned delay, error code, version number, connection number, CPU / memory / disk usage, replication delay, lock waiting, number of slow queries and other key indicators, and then determines whether the node of the database instance is alive, whether a master-slave switchover occurs, whether there is a read-only lock, memory overflow, thread pool depletion and other abnormal events based on these key indicators, and finally aggregates to generate a multi-dimensional fault detection result and running state information.

[0088] S203, analyze and calculate the fault detection result and running state information of each database instance to obtain an analysis and calculation result.

[0089] In this embodiment, after the fault detection result and running state information of each database instance are acquired, the collected original heartbeat delay, error code, connection number, CPU / memory / disk usage, replication delay, lock waiting, number of slow queries and other time series data can be preprocessed (the preprocessing methods include cleaning, normalization and missing value processing), and then the preprocessed data is aggregated according to the time window by using a streaming computing engine, the mean, variance, percentile, same-period and same-ratio, SLA compliance rate, health score and other information of the preprocessed data are calculated, and then the abnormal fluctuations, potential faults and trend turning points are detected and predicted in real time by combining the rule engine and machine learning algorithms (such as isolated forest, LSTM and XGBoost), to generate an analysis and calculation result including availability level, capacity bottleneck, performance degradation and fault root cause probability distribution.

[0090] S204, determining whether a preset fault type exists in the database instance based on the analysis and calculation result.

[0091] The preset fault type includes node failure, instance failure, slow query overload, and excessively large master-slave delay.

[0092] In this embodiment, after the analysis and calculation result is acquired, it can be determined based on the analysis and calculation result whether the fault type of the database instance is at least one of node failure, instance failure, slow query overload, and excessively large master-slave delay.

[0093] S205, if the preset fault type exists, performing a self-recovery task corresponding to the preset fault type.

[0094] The self-recovery task corresponding to the node failure includes: deploying an architecture of a single node or a replica set on the node at the same time, and loading context information on the node, so as to pull up the node for recovery when the node executes a start command.

[0095] The self-recovery task corresponding to the instance failure includes: deploying an architecture of a shard cluster on the node at the same time, classifying and sorting the types of the node, and then performing a recovery operation in the order of a configuration node, a data node, and a routing node.

[0096] The self-recovery task corresponding to the slow query overload includes: searching for a corresponding table and a field on the node, analyzing whether a corresponding index of the corresponding table and the field is not established through a search result, and issuing a warning when the corresponding index of the corresponding table and the field is not established.

[0097] The self-recovery task corresponding to the large master-slave delay includes: issuing a warning, and using a read-write separation or a majority confirmation and re-writing scheme when the oplog window size is not exceeded, and indicating that the standby node cannot be recovered when the oplog window size is exceeded, so that the node can be rebuilt through hot backup data.

[0098] In the embodiment, if the preset failure type exists in the failure type of the database instance, the self-recovery task corresponding to the preset failure type is executed to process the failure of the node of the database instance.

[0099] In the embodiment, in a scenario of needing to manage a large number of MongoDB instances, based on the node information of each database instance, the node of each database instance is initiated for failure detection, the running state with a failure trend is calculated and analyzed, so that each database instance can autonomously detect and judge the failure type, and perform a corresponding failure self-recovery task, which can not only reduce the mistakes and delay of manual intervention operation, but also reduce the probability of business system failure and shorten the recovery time.

[0100] In the embodiment, the detailed process of obtaining the failure detection result and the running state information of each database instance is explained based on the embodiment shown in Figure 2 In an exemplary embodiment, as shown in Figure 3 S202 includes:

[0101] S301, based on the IP address and the port in the node information, performing network connectivity testing and heartbeat detection on the node of each database instance to confirm whether the node is online.

[0102] The connectivity testing can detect the network connection of the node of each database instance to ensure that the network communication between the nodes of each database instance has no problem.

[0103] The heartbeat detection can determine the availability of the node of each database instance to confirm whether the node of each database instance is online. In addition, the state of the node of each database instance can be obtained through the db.serverStatus() command.

[0104] In this embodiment, the IP address and the port of each database instance can be obtained first, and then the network connectivity test and the heartbeat detection can be initiated to the node of each database instance according to the IP address and the port to determine whether the node of each database instance is online.

[0105] S302, a connection is established to the node determined to be online, and a state query command is executed to obtain the process survival state and the running state data of the node.

[0106] In this embodiment, if it is determined according to the results of the connectivity test and the heartbeat detection that the node corresponding to the database instance is online, the node of the online database instance can be connected to the management center, and the state query command of the node of the online database instance can be executed on the management center to obtain the process survival state and the running state data of the node of the online database instance.

[0107] S303, the results of the network connectivity test failure, the heartbeat detection non-response, and the state query command execution exception are included in the fault detection results.

[0108] In this embodiment, if the results of the connectivity test and the heartbeat detection of the node of the database instance by the management center are at least one of the network connectivity test failure, the heartbeat detection non-response, and the state query command execution exception, at least one of the network connectivity test failure, the heartbeat detection non-response, and the state query command execution exception is included in the fault detection results.

[0109] S304, the online state, the process survival state, and the detailed running state data of the node of each database instance are included in the running state information.

[0110] In this embodiment, if the results of the connectivity test and the heartbeat detection of the node of the database instance by the management center are that the node exists, the online state, the process survival state, and the running state data of the node obtained by the state query command are included in the running state information.

[0111] In this embodiment, the scheme first quickly screens out unreachable nodes through IP address and port level network connectivity test and heartbeat detection, then establishes connection with online nodes and performs state query, accurately identifies process live or dead and load details, and records network failure, heartbeat timeout and command exception into fault detection results, and deposits online state, process health and running index into state information. In addition, through one scan, a high confidence availability image can be output in layers, significantly reducing false positives and false negatives, shortening fault positioning time, providing real-time, fine and quantifiable factual basis for automatic disaster recovery and elastic scheduling, reducing business interruption and operation and maintenance cost.

[0112] In the above Figure 2 embodiment, in the case where the preset fault type includes an instance fault, the detailed process of judging whether the database instance has the preset fault type can be explained and described. In an exemplary embodiment, as shown in Figure 4 S204 includes:

[0113] S401, when the deployment architecture of the database instance is a sharded cluster, based on the analysis calculation result, the state data of any one of the routing node, the configuration node and each shard data node of the database instance is counted.

[0114] The state data of the routing node includes the summary connection pool usage rate of the routing node, the routing table version, the forwarding delay and the error rate, and measures the entry load and routing consistency.

[0115] The state data of the configuration node includes tracking the cluster metadata read-write delay, election state, chunk distribution deviation and lock waiting of the configuration node, and ensures that the configuration information is authoritative and highly available.

[0116] The state data of the shard data node includes the master-slave delay within the shard, the disk remaining capacity, the chunk quantity and the migration queue length, QPS / CPU / memory hotspots of each shard data node, and accurately locates data skew or performance bottleneck

[0117] In this embodiment, when the deployment architecture of the database instance is a sharded cluster, based on the analysis calculation result, the system can count the exclusive state data of the routing node, the configuration node and each shard data node, then aggregate the key indicators of the three types of nodes, and form a panoramic health report covering the routing layer, the metadata layer and the data layer, providing layer-by-layer drillable decision basis for balanced sharding, elastic scaling and fault switching.

[0118] S402, if the state data meets the target condition, it is determined that the database instance has an instance fault; the state data meeting the target condition includes any of the following: all routing nodes are offline or unavailable; the configuration node has no master node; any shard data node has no master node.

[0119] In this embodiment, if the state data obtained above meets any of the following conditions: all routing nodes are offline or unavailable; the configuration node has no master node; any shard data node has no master node, it is determined that the database instance has an instance fault.

[0120] In this embodiment, by locking the routing full hanging, the configuration without master, and the shard without master, the cluster-level fault is accurately determined, the false alarm of single-point jitter is avoided, and the switching and repair time is significantly shortened.

[0121] In the above Figure 4 based on the embodiment shown in the above Figure 5 , the S204 includes:

[0122] S501, based on the analysis calculation result, a database system configuration file query command is executed to extract slow query logs of each database instance.

[0123] In this embodiment, when the analysis calculation result is obtained, slow query detection can be performed based on the analysis calculation result, and slow query data is collected. The slow query data will be recorded in the capped collection of system.profile. The capped collection set will record new slow query data in a loop according to the pre-set size limit.

[0124] Further, the capped collection set can be queried using the database system configuration file query command db.system.profile.find(). The query statement with an execution time exceeding the threshold value is obtained to obtain the slow query log.

[0125] S502, the extracted slow query log is classified by hash according to the query statement characteristics, and the occurrence frequency of each type of slow query is counted.

[0126] In this embodiment, when the extracted slow query log is obtained, the extracted slow query log can be classified by hash according to the query statement characteristics to count the occurrence frequency of each type of slow query.

[0127] S503. If the frequency of slow queries of the target class exceeds the preset frequency threshold, and time series analysis confirms that the frequency of slow queries of the target class is gradually increasing, then it is determined that the database instance corresponding to the slow query of the target class has slow query overload.

[0128] In this embodiment, if a target type slow query is found among the occurrence frequencies of multiple types of slow queries, and the occurrence frequency of this target type slow query exceeds a preset frequency threshold, then a time series analysis is performed on the target type slow query. If the time series analysis result of the target type slow query shows that the occurrence frequency of the target type slow query is gradually increasing, then it is determined that the database instance corresponding to the target type slow query has a slow query overload.

[0129] In this embodiment, by automatically pulling slow logs, hashing and merging similar SQL queries, and overlaying time-series analysis, the system can accurately identify slow query categories with frequencies exceeding thresholds and continuously increasing within minutes, locate potential overload sources in advance, avoid database performance drops sharply due to slow query accumulation, and gain a golden processing window for subsequent index optimization, SQL rewriting, or resource expansion.

[0130] In this embodiment, in the above Figure 2 Based on the illustrated embodiment, the detailed process for determining whether a database instance exhibits a preset fault type can be explained when the preset fault type includes excessive primary / standby latency. In an exemplary embodiment, such as Figure 6 As shown, the above S204 includes:

[0131] S601. When the database instance is deployed in a replica set architecture, based on the analysis and calculation results, the operation timestamp information of the master node and slave node is obtained by executing the replica set status query command, and the master-slave latency is calculated.

[0132] The rs.status() command, which queries the status of a replica set, provides detailed status information about the replica set, including the latency of each replica.

[0133] In this embodiment, when the database instance is deployed in a replica set architecture, master-slave latency detection can be performed based on the analysis and calculation results. The operation timestamp information between the master node and the slave node can be obtained by executing the replica set status query command rs.status(), so as to calculate the master-slave latency based on the operation timestamp information between the master node and the slave node.

[0134] S602. If the primary / standby delay exceeds the preset time threshold, and timing analysis confirms that the primary / standby delay is gradually increasing, execute the replica set replication information query command to query the window size of the replica set operation log.

[0135] In the embodiment, after the master-slave delay is obtained, it can be determined whether the master-slave delay exceeds the preset time threshold. If the master-slave delay exceeds the preset time threshold, the master-slave delay can be analyzed in time sequence. If the result of the time sequence analysis is that the master-slave delay shows a gradually increasing trend, a replica set replication information query command rs.printReplicationInfo () is executed to query the window size of the operation log of the replica set.

[0136] In the embodiment, by comparing the delay and the window size of the operation log of the replica set, it can be determined whether the secondary node has failed to catch up with the primary node. In the replica set architecture of the database MongoDB, the secondary node of the replica set lags behind the window size of the operation log of the primary node of the replica set, but has not exceeded this range. Therefore, the real-time read-write separation strategy or ensuring that the write operation obtains the confirmation of the majority of nodes before being considered successful can be used.

[0137] In S603, if the master-slave delay has exceeded the window size of the operation log, it is determined that the database instance has a master-slave delay too large fault.

[0138] In the embodiment, if the master-slave delay of a certain secondary node exceeds the window size of the operation log and cannot be restored to the latest state through the regular replication process, it is determined that the database instance has a master-slave delay too large fault. In this case, the latest valid data hot backup is used to rebuild the node, and data is copied from the primary node or other healthy backup nodes to restore the state of the node, so that it can rejoin the replica set and continue to participate in data replication and distribution.

[0139] In the embodiment, under the replica set architecture, the master-slave delay is calculated in real time by periodically executing the replica set state query command to capture the operation time stamps of the primary and secondary nodes, and the time sequence trend analysis is superimposed. The system can issue a warning as soon as the delay breaks through the threshold and continues to rise. Then, the window size of the operation log is automatically obtained. Once it is confirmed that the delay is greater than the window, it is immediately determined that the master-slave delay is too large, and the alarm or automatic expansion is triggered in advance to avoid the secondary node from entering full resynchronization due to failing to catch up with the operation log, thereby significantly reducing the risk of data loss and business read-only interruption.

[0140] In the embodiment, after the master-slave delay is obtained, it can be determined whether the master-slave delay exceeds the preset time threshold. If the master-slave delay exceeds the preset time threshold, the master-slave delay can be analyzed in time sequence. If the result of the time sequence analysis is that the master-slave delay shows a gradually increasing trend, a replica set replication information query command rs.printReplicationInfo () is executed to query the window size of the operation log of the replica set. Figure 2 Based on the embodiment shown in Figure 7 , S205 includes:

[0141] S701. If the preset fault type is node fault and the database instance deployment architecture is a single node or replica set, the self-recovery tasks to be executed include: loading the context information of the faulty node, calling the database node startup command, and waiting for the node process to complete initialization.

[0142] In this embodiment, if the preset fault type is node fault and the database instance deployment architecture is a single node or replica set, the most recent complete and verified hot backup data can be located first. Then, the hot backup data is pulled from the faulty slave node locally or from the nearest temporary instance to complete the full data recovery. The master node or another healthy slave node with the lowest latency is used as the synchronization source to start incremental pulling, fill in the differences in operation logs after the backup point, and then perform operations such as replSetReconfig offline, clearing the local dbpath, and reinitializing on the faulty slave node. After the initial synchronization is completed, it is automatically added back to the replica set and voting and read permissions are restored. During the above process, its lag window can be continuously monitored. If the latency falls back to within the threshold, the self-healing is marked as successful; otherwise, the alarm is upgraded and handed over to manual review.

[0143] S702. If the preset fault type is instance fault and the database instance is deployed in a sharded cluster architecture, the self-recovery task to be executed includes: classifying and sorting the faulty nodes in the order of configuration nodes, sharded data nodes, and routing nodes, and executing the corresponding recovery subtask flow for each type of faulty node.

[0144] In this embodiment, if the preset fault type is instance fault and the database instance is deployed in a sharded cluster architecture, the faulty nodes can first be sorted by role. The sorting order can include: configuration nodes, sharded data nodes, and routing nodes, to ensure priority recovery of the metadata layer. Then, if the configuration node fails, a new database instance is immediately launched in another availability zone, members are rebuilt using the most recent snapshot and incremental operation logs, the sharded cluster's config RS is updated, and the chunk mapping integrity is verified. If the sharded data node fails, and the faulty node is the master node, a new master node election is triggered; if the faulty node is a slave node, recovery is performed according to the sub-process of excessive master-slave latency (including: pulling hot backup data, incremental synchronization, and re-adding to the replica set). Next, if the routing node (mongos) fails (i.e., the routing node is stateless), a new instance is directly restarted within seconds based on the container / virtual machine template, the configuration node address is automatically mounted, and it is added to the load balancing pool. Finally, during the above process, after each type of role recovery is completed, a cluster-level health check (including: config...) is immediately performed. If consistency, chunk balancing, and routing table refresh all pass, the self-healing process is marked as successful; otherwise, subsequent steps are stopped and the alarm is upgraded.

[0145] S703, if the preset fault type is master-slave delay too large, and the deployment architecture of the database instance is a data node replica set in a replica set or a shard cluster, the self-recovery task performed includes: obtaining the latest valid hot backup data of the database instance; copying data from the master node or the healthy backup node to the fault slave node; performing initialization operation on the fault slave node, and rejoining the initialized slave node to the replica set.

[0146] In this embodiment, if the preset fault type is master-slave delay too large, and the deployment architecture of the database instance is a data node replica set in a replica set or a shard cluster, the latest complete and available hot backup data of the database instance can be extracted from the backup system first to ensure data integrity and consistency, then the missing or lagging data is copied to the fault slave node with the master node or the healthy slave node with the lowest delay as the data source to ensure data synchronization, and then the old data copy of the fault slave node is cleared, the data directory and replica set configuration of the fault slave node are reinitialized, the initialized slave node is re-joined to the replica set to restore its normal role and function, and the synchronization state of the initialized slave node with the master node is verified. Finally, the delay of the node is continuously monitored to confirm that the node has recovered to the normal synchronization state, if the delay falls within the set threshold, the self-healing is marked as completed, otherwise the manual intervention process is triggered.

[0147] S704, if the preset fault type is slow query overload, the self-recovery task performed includes: retrieving the database table and field corresponding to the slow query overload; analyzing whether there is no matching index established for the database table and field; generating a recommended strategy for establishing target index and issuing a warning; synchronizing the slow query analysis result and the recommended strategy to the log system.

[0148] In this embodiment, if the preset fault type is slow query overload, the slow SQL whose time consumption exceeds the threshold value can be captured in real time, the database, table and field combination accessed by the slow SQL are parsed, the high-frequency hot table and filtering / sorting / connection column are located, and then the optimizer explain and statistical information are called to check whether there is a suitable index for these fields one by one. If a missing, redundant or poor selectivity index is found, it is marked as unmatched, and then based on the field cardinality, query frequency, data distribution and write load, an optimal index script is automatically generated, the creation cost is evaluated, and finally the complete slow query text, execution plan, missing index details, recommended index DDL, risk level and timestamp are written into the log and audit library for subsequent audit, automatic index drilling platform or capacity report subscription consumption.

[0149] In this embodiment, through the combination of the four types of fault scenarios, i.e., node, instance, delay, and slow query, and the architecture-aware self-recovery strategy, the system can automatically complete node restart, sequential reconstruction of a shard cluster, master-slave delay flattening, and index missing repair within minutes, significantly reducing the rate of manual intervention and the length of business interruption, while depositing each diagnosis conclusion and optimization suggestion as an auditable and replayable log asset, providing continuous data support for subsequent capacity planning, performance baseline improvement, and cost optimization.

[0150] In this embodiment, the detailed process after the node of the database instance has the preset fault type and the self-recovery task corresponding to the preset fault type is executed is explained based on the embodiment shown in Figure 2 Figure 8 In this embodiment, the method further includes:

[0151] S206, for the database instance node after the recovery, reinitiate the fault detection and running state monitoring, and obtain the fault detection result and running state information after the recovery.

[0152] In this embodiment, after the recovery action of the database instance node is completed, the system immediately starts a secondary health check on the database instance node, including sending four types of probes, i.e., connection, read-write, replication delay, and load, for two detection periods, collecting key indicators such as process state, master-slave delay, and slow query count, generating a post-recovery snapshot, and comparing with the baseline, if the comparison result is that all indicators fall within the threshold, marking the self-recovery as successful and closing the alarm, otherwise, immediately triggering the escalation channel, notifying manual review, and writing the detection result back to the audit library for subsequent fault trend analysis and self-recovery strategy optimization.

[0153] S207, based on the fault detection result and running state information after the recovery, verifying whether the self-recovery condition is met.

[0154] In this embodiment, after obtaining the fault detection result and running state information after the recovery, the system can automatically compare with the preset self-recovery success threshold piece by piece: whether the node is connectable, whether the master-slave delay is less than N seconds, whether the TPS / QPS is restored to the 80% interval before the fault, and whether the number of slow queries is not higher than 1.2 times the baseline, if all the above conditions are met, it is determined that the self-recovery is effective and the work order is closed, if any indicator does not meet the standard, the self-recovery mark is rolled back, the alarm escalation is triggered, and a difference report is generated.

[0155] S208, if the verification is passed, it is determined that the fault self-recovery process is completed, if the verification fails, the self-recovery task corresponding to the preset fault type is re-determined, and the fault self-recovery process is executed again.

[0156] ​In this embodiment, if the check passes, a self-healing success event is automatically generated, the alarm is closed, the resource lock is released, and it is determined that the fault self-recovery process is completed; if the check fails, the label in the node self-healing is immediately rolled back, the fault type is re-matched according to the latest snapshot (including: master-slave delay being too large, node fault, instance fault, and slow query overload), a new round of repair script (for example, source replacement and resynchronization, promotion of a new master, index supplement, or instance expansion) is generated, and the self-healing pipeline is entered again until the success threshold is reached or the cumulative number of failures reaches a certain number, and then forced manual intervention is performed.

[0157] Further, the embodiment also provides a check process of a node of a database instance after fault recovery, including:

[0158] If it is node fault recovery, it is checked whether a first self-recovery condition is met, the first self-recovery condition including: network connectivity of the node, process survival, and normal execution of a status query command;

[0159] If it is instance fault recovery, it is checked whether a second self-recovery condition is met, the second self-recovery condition including: a sharding cluster meeting at least one surviving routing node, a configuration node having a master node, and each shard having a master node;

[0160] If it is master-slave delay being too large recovery, it is checked whether a third self-recovery condition is met, the third self-recovery condition including: a reestablished slave node normally synchronizing master node data, and the master-slave delay being reduced to within a preset safety threshold.

[0161] In this embodiment, through a closed-loop mechanism of recovery being detection and detection being check, the system sets first, second, and third self-recovery conditions for three types of scenarios of nodes, instances, and master-slave delay, respectively, verifies core indexes such as network, process, master-slave role, sharding topology, and delay at a second level after completion of self-healing actions; if the check passes once, the alarm is automatically closed, the lock is released, and the work order is ended, otherwise, the fault type is immediately re-matched, the repair script is re-run, until the threshold is reached or manual intervention is upgraded, thereby reducing false positives and repeated fault rates to the minimum, realizing truly unattended, observable, and rollbackable continuous self-healing, significantly shortening MTTR, and depositing credible recovery baseline data.

[0162] In one embodiment, referring to Figure 9 A fault processing method of a database is also provided, including:

[0163] T1, storing node information of each database instance in a target resource pool; the node information includes an IP address and a port;

[0164] T2, performing network connectivity testing and heartbeat detection on nodes of each database instance based on the IP address and the port in the node information, to confirm whether the nodes are online;

[0165] T3, establish a connection with the node detected as online and execute a state query command to obtain the process alive state and running state data of the node;

[0166] T4, incorporate the results of network connectivity testing failure, heartbeat detection non-response, and state query command execution exception into the fault detection results;

[0167] T5, incorporate the online state, process alive state, and detailed running state data of the nodes of each database instance into the running state information;

[0168] T6, analyze and calculate the fault detection results and running state information of each database instance to obtain analysis and calculation results;

[0169] T7, based on the analysis and calculation results, determine whether the database instance has a preset fault type; the preset fault type includes: instance fault, node fault, slow query overload, and large master-slave delay;

[0170] T8, if there is a preset fault type, if the preset fault type is a node fault, and the deployment architecture of the database instance is a single node or a replica set, the self-recovery tasks performed include: loading the context information of the fault node, calling the database node startup command, and waiting for the node process to complete initialization;

[0171] T9, if the preset fault type is an instance fault, and the deployment architecture of the database instance is a shard cluster, the self-recovery tasks performed include: sorting the fault nodes in the order of configuration nodes, shard data nodes, and routing nodes, and executing corresponding recovery sub-tasks for each type of fault node;

[0172] T10, if the preset fault type is a large master-slave delay, and the deployment architecture of the database instance is a replica set or a data node replica set in a shard cluster, the self-recovery tasks performed include: obtaining the latest valid hot backup data of the database instance; copying data from the master node or the healthy backup node to the fault slave node; performing initialization operations on the fault slave node, and rejoining the initialized slave node to the replica set;

[0173] T11, if the preset fault type is slow query overload, the self-recovery tasks performed include: retrieving the database tables and fields corresponding to slow query overload; analyzing whether there is no matching index established for the database tables and fields; generating a recommended strategy for establishing target indexes and issuing a warning; synchronizing the slow query analysis results and recommended strategies to the log system;

[0174] T12, reinitiate fault detection and running state monitoring for the database instance nodes after recovery, and obtain the fault detection results and running state information after recovery;

[0175] T13, verifying whether the self-recovery condition is met based on the fault detection result after recovery and the running state information;

[0176] T14, if the verification passes, determining that the fault self-recovery process is completed; if the verification fails, re-determining the self-recovery task corresponding to the preset fault type, and executing the fault self-recovery process again.

[0177] It should be noted that the description in T1-T14 above can refer to the related description in the above embodiments, and the effects are similar, and the present embodiment will not be repeated here.

[0178] It should be understood that although each step in the flowchart involved in each of the above embodiments is displayed in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise stated herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in each of the above embodiments can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately executed with other steps or steps or stages in other steps.

[0179] Based on the same inventive concept, the embodiments of the present application also provide a database fault processing apparatus for implementing the above-mentioned database fault processing method. The implementation scheme for solving the problem provided by the apparatus is similar to the implementation scheme described in the above method, so the specific limitations in one or more database fault processing apparatus embodiments provided below can refer to the limitations of the database fault processing method in the above text, and will not be repeated here.

[0180] In one exemplary embodiment, as shown in Figure 10 a database fault processing apparatus is provided, comprising: a storage module 10, a monitoring module 11, an analysis module 12, a judgment module 13 and an execution module 14, wherein:

[0181] The storage module 10 is configured to store node information of each database instance in the target resource pool; the node information includes IP address and port.

[0182] The monitoring module 11 is configured to initiate fault detection and running state monitoring to the nodes of each database instance in the target resource pool based on the node information, to obtain fault detection results and running state information of each database instance.

[0183] The analysis module 12 is configured to analyze and calculate the fault detection result and the running state information of each database instance to obtain an analysis result.

[0184] The judgment module 13 is configured to judge whether the database instance has a preset fault type based on the analysis result.

[0185] The execution module 14 is configured to execute a self-recovery task corresponding to the preset fault type when the preset fault type exists.

[0186] In an exemplary embodiment, the monitoring module 11 includes:

[0187] The detection unit is specifically configured to perform network connectivity testing and heartbeat detection on the nodes of each database instance based on the IP address and the port in the node information to confirm whether the nodes are online.

[0188] The execution unit is specifically configured to establish a connection to the nodes confirmed to be online and execute a state query command to obtain process survival state and running state data of the nodes.

[0189] The first inclusion unit is specifically configured to include the results of network connectivity testing failure, heartbeat detection non-response, and state query command execution exception in the fault detection result.

[0190] The second inclusion unit is specifically configured to include the online state, process survival state, and detailed running state data of the nodes of each database instance in the running state information.

[0191] In an exemplary embodiment, the preset fault type includes instance failure, and the judgment module 13 includes:

[0192] The first statistical unit is specifically configured to, when the deployment architecture of the database instance is a sharded cluster, statistically analyze the state data of any one of the routing nodes, the configuration nodes, and the shard data nodes of the database instance based on the analysis result.

[0193] The first determination unit is specifically configured to determine that the database instance has instance failure if the state data meets a target condition, and the state data meeting the target condition includes any one of the following: all routing nodes are offline or unavailable; the configuration node has no master node; any shard data node has no master node.

[0194] In an exemplary embodiment, the preset fault type includes slow query overload, and the judgment module 13 includes:

[0195] The execution unit is specifically configured to execute a database system configuration file query command based on the analysis result to extract the slow query log of each database instance.

[0196] The second statistical unit is specifically configured to hash and classify the extracted slow query logs according to query statement features, and count the occurrence frequency of each type of slow query.

[0197] The second determination unit is specifically configured to determine that the database instance corresponding to the target type of slow query exists slow query overload if the occurrence frequency of the target type of slow query exceeds the preset frequency threshold and the time series analysis confirms that the occurrence frequency of the target type of slow query shows a gradually increasing trend.

[0198] In an exemplary embodiment, the preset fault type includes that the master-slave delay is too large, and the determination module 13 includes:

[0199] The calculation unit is specifically configured to, when the deployment architecture of the database instance is a replica set, acquire operation timestamp information of the master node and the slave node by executing a replica set state query command based on the analysis calculation result, and calculate the master-slave delay.

[0200] The query unit is specifically configured to execute a replica set replication information query command to query the window size of the operation log of the replica set when the master-slave delay exceeds the preset time threshold and the time series analysis confirms that the master-slave delay shows a gradually increasing trend.

[0201] The third determination unit is specifically configured to determine that the database instance exists the fault of the master-slave delay being too large if the master-slave delay has exceeded the window size of the operation log.

[0202] In an exemplary embodiment, the execution module 14 includes:

[0203] The first execution unit is specifically configured to, if the preset fault type is node fault and the deployment architecture of the database instance is a single node or a replica set, the self-recovery task executed includes loading context information of the fault node, calling a database node start command, and waiting for the node process to complete initialization.

[0204] The second execution unit is specifically configured to, if the preset fault type is instance fault and the deployment architecture of the database instance is a shard cluster, the self-recovery task executed includes sorting the fault nodes in the order of configuration nodes, shard data nodes, and routing nodes, and executing a corresponding recovery subtask flow for each type of fault node.

[0205] The third execution unit is specifically configured to, if the preset fault type is that the master-slave delay is too large and the deployment architecture of the database instance is a replica set or a data node replica set in a shard cluster, the self-recovery task executed includes acquiring the latest valid hot backup data of the database instance, copying data from the master node or the healthy backup node to the fault slave node, and performing initialization operation on the fault slave node to rejoin the replica set after the initialization of the slave node.

[0206] The fourth execution unit is specifically configured to, if the preset fault type is a slow query overload, the self-recovery task executed includes: retrieving a database table and a field corresponding to the slow query overload; analyzing whether the database table and the field exist in a case where a matching index is not established; generating a recommended strategy for establishing a target index and issuing a warning; and synchronously recording a slow query analysis result and the recommended strategy to a log system.

[0207] In an exemplary embodiment, the apparatus further includes:

[0208] The obtaining module is configured to, for the database instance node after the execution recovery, re-initiate fault detection and running state monitoring, and obtain a fault detection result and running state information after the recovery;

[0209] The checking module is configured to check whether a self-recovery condition is met based on the fault detection result and the running state information after the recovery;

[0210] The execution module is configured to, if the checking passes, determine that a fault self-recovery process is completed; and if the checking fails, re-determine a self-recovery task corresponding to the preset fault type and execute the fault self-recovery process again;

[0211] If the node fault recovery is performed, the checking is whether a first self-recovery condition is met, the first self-recovery condition including: a node network is connected, a process is alive, and a state query command is executed normally; if the instance fault recovery is performed, the checking is whether a second self-recovery condition is met, the second self-recovery condition including: a sharding cluster meets a condition that at least one alive routing node, a configuration node has a master node, and each shard has a master node; and if the master-backup large-delay recovery is performed, the checking is whether a third self-recovery condition is met, the third self-recovery condition including: a reestablished slave node normally synchronizes master node data, and a master-backup delay is reduced to within a preset safety threshold.

[0212] The modules in the database fault processing apparatus can be all or partially implemented by software, hardware, and a combination thereof. The modules can be embedded in or independent of a processor in a computer device in a hardware form, or stored in a memory in a computer device in a software form, so as to be called and executed by a processor to perform operations corresponding to the modules.

[0213] In an embodiment, a computer device is also provided, including a memory and a processor, the memory storing a computer program, and the processor implementing steps in the above method embodiments when executing the computer program.

[0214] In an embodiment, a computer readable storage medium is provided, storing a computer program, and the computer program implementing steps in the above method embodiments when executed by a processor.

[0215] In an embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of any of the above method embodiments.

[0216] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned method embodiments. Any reference to a memory, database or other medium used in the embodiments provided in the present application can include at least one of a non-volatile memory and a volatile memory. The non-volatile memory can include a read-only memory (ROM), a magnetic tape, a floppy disk, a flash memory, an optical storage, a high-density embedded non-volatile memory, a resistive random access memory (ReRAM), a magnetoresistive random access memory (MRAM), a ferroelectric random access memory (FRAM), a phase change memory (PCM), a graphene memory, etc. The volatile memory can include a random access memory (RAM) or an external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0217] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0218] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A database fault handling method, characterized in that, Applied to a control center, the method includes: Stores node information for each database instance within the target resource pool; the node information includes IP address and port. Based on the node information, fault detection and operation status monitoring are initiated on the nodes of each database instance in the target resource pool to obtain the fault detection results and operation status information of each database instance. The fault detection results and operating status information of each database instance are analyzed and calculated to obtain the analysis and calculation results; Based on the analysis and calculation results, it is determined whether the database instance has a preset fault type; If a preset fault type exists, then the self-recovery task corresponding to the preset fault type is executed.

2. The method according to claim 1, characterized in that, Based on the node information, the step of initiating fault detection and operational status monitoring to the nodes of each database instance within the target resource pool to obtain the fault detection results and operational status information of each database instance includes: Based on the IP address and port in the node information, network connectivity tests and heartbeat checks are performed on the nodes of each database instance to confirm whether the nodes are online. Establish a connection to the nodes that are detected as online and execute a status query command to obtain the process liveness and running status data of the nodes; The results of the network connectivity test failure, the heartbeat detection failure, and the status query command execution error are included in the fault detection results; The online status of nodes in each database instance, the survival status of the process, and detailed running status data are included in the running status information.

3. The method according to claim 1, characterized in that, The preset fault types include instance faults; determining whether the database instance has a preset fault type based on the analysis and calculation results includes: When the deployment architecture of the database instance is a sharded cluster, based on the analysis and calculation results, the status data of any one of the routing nodes, configuration nodes, and sharded data nodes of the database instance are statistically analyzed. If the status data meets the target conditions, then the database instance is determined to have an instance failure. The state data satisfying the target condition includes any one of the following: All routing nodes are offline or unavailable; Configure the node to have no master node; No single data shard node has a master node.

4. The method according to claim 3, characterized in that, The preset fault types include slow query overload; the step of determining whether the database instance has a preset fault type based on the analysis and calculation results includes: Based on the analysis and calculation results, execute the database system configuration file query command to extract the slow query logs of each database instance; The extracted slow query logs are categorized by hash based on query statement characteristics, and the frequency of occurrence of each type of slow query is statistically analyzed. If the frequency of a target type slow query exceeds a preset frequency threshold, and time series analysis confirms that the frequency of the target type slow query is gradually increasing, then it is determined that the database instance corresponding to the target type slow query is experiencing slow query overload.

5. The method according to claim 1, characterized in that, The preset fault types include excessive primary / standby latency; the step of determining whether the database instance has a preset fault type based on the analysis and calculation results includes: When the database instance is deployed in a replica set architecture, based on the analysis and calculation results, the operation timestamp information of the master node and slave node is obtained by executing the replica set status query command, and the master-slave latency is calculated. If the primary / standby delay exceeds a preset time threshold, and timing analysis confirms that the primary / standby delay is gradually increasing, execute a replica set replication information query command to query the window size of the replica set's operation log. If the master-slave latency exceeds the window size of the operation log, then the database instance is determined to have a master-slave latency too high fault.

6. The method according to claim 1, characterized in that, The execution of the self-recovery task corresponding to the preset fault type includes: If the preset fault type is a node fault, and the database instance is deployed in a single-node or replica set architecture, the self-recovery tasks to be performed include: loading the context information of the faulty node, calling the database node startup command, and waiting for the node process to complete initialization. If the preset fault type is an instance fault and the database instance is deployed in a sharded cluster architecture, the self-recovery task to be executed includes: classifying and sorting the faulty nodes in the order of configuration nodes, sharded data nodes, and routing nodes, and executing the corresponding recovery subtask flow for each type of faulty node; If the preset fault type is a primary / standby delay that is too large, and the deployment architecture of the database instance is a replica set or a data node replica set in a sharded cluster, then the self-recovery task to be performed includes: obtaining the most recent valid hot backup data of the database instance; copying the data from the primary node or healthy backup node to the faulty slave node; performing an initialization operation on the faulty slave node, and re-adding the initialized slave node to the replica set; If the preset fault type is slow query overload, the self-recovery tasks performed include: retrieving the database table and fields corresponding to the slow query overload; analyzing whether there are any cases where no matching index has been created in the database table and fields; generating a recommended strategy for creating the target index and issuing an early warning; and synchronously recording the slow query analysis results and recommended strategies to the log system.

7. The method according to claim 1, characterized in that, If a preset fault type exists, after executing the self-recovery task corresponding to the preset fault type, the method further includes: For the database instance nodes that have been restored, re-initiate fault detection and runtime status monitoring to obtain the fault detection results and runtime status information after the restoration. Based on the recovered fault detection results and operating status information, verify whether the self-recovery conditions are met; If the verification passes, the fault self-recovery process is considered complete; if the verification fails, a new self-recovery task corresponding to the preset fault type is determined, and the fault self-recovery process is executed again. If it is a node failure recovery, then it is checked whether the first self-recovery condition is met. The first self-recovery condition includes: the node network is connected, the process is alive and the status query command is executed normally. If it is an instance failure recovery, then check whether the second self-recovery condition is met. The second self-recovery condition includes: the sharded cluster has at least one live routing node, the configuration node has a master node, and each shard has a master node. If the master-slave delay is too large during recovery, then check whether the third self-recovery condition is met. The third self-recovery condition includes: the rebuilt slave node normally synchronizes the master node data, and the master-slave delay is reduced to within a preset safety threshold.

8. A database fault handling device, characterized in that, The device includes: The storage module is used to store node information of each database instance in the target resource pool, and the node information includes IP address and port. The detection module is used to initiate fault detection and operation status monitoring to the nodes of each database instance in the target resource pool based on the node information, so as to obtain the fault detection results and operation status information of each database instance. The analysis and calculation module is used to analyze and calculate the fault detection results and operating status information of each database instance to obtain the analysis and calculation results; The fault diagnosis module is used to determine whether the database instance has a preset fault type based on the analysis and calculation results; The self-recovery module is used to assemble a task flow corresponding to a preset fault type if a preset fault type exists, and to execute a fault self-recovery process based on the task flow.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images