Database fault processing method and device, computer equipment and readable storage medium

Abstract

The invention relates to a database fault processing method and device, computer equipment and a readable storage medium. The method comprises the steps that operation data of multi-level objects to be detected in a database are obtained through a multi-level state machine, each level of the multi-level state machine corresponds to each level of the multi-level objects to be detected in a one-to-one mode, and the state machine of each level is used for obtaining the operation data of the objects to be detected of the corresponding level; according to the operation data of the multi-level to-be-detected objects, determining the operation state of the to-be-detected object of each level; when the operation state represents that the to-be-detected object is in a fault state, determining a fault recovery strategy corresponding to the fault state; and repairing the to-be-detected object in the fault state according to the fault recovery strategy. By adopting the method, the database fault detection and recovery efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a database fault processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] Traditional distributed databases often consist of multiple components and multiple nodes to store and manage data. When a node fails, if it cannot be detected and processed in time, it may cause data loss or service interruption. Therefore, it is crucial to quickly and effectively identify the failure and restore the service.

[0003] However, due to the diversity of failure causes, traditional database systems face many challenges in handling failures. Usually, the response of database systems to specific failures depends on fixed recovery strategies or requires manual intervention, resulting in low efficiency in handling failures. Summary of the invention

[0004] The embodiments of the present application provide a database fault processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product, which can improve the efficiency of database fault processing.

[0005] In one aspect, the present application provides a method for handling database failures. The method comprises:

[0006] Acquire the operation data of the multi-level objects to be detected in the database through a multi-level state machine, wherein each level of the multi-level state machine corresponds to each level of the multi-level objects to be detected, and the state machine of each level is used to acquire the operation data of the objects to be detected at the corresponding level;

[0007] Determining the operating status of the objects to be detected at each level according to the operating data of the objects to be detected at the multiple levels;

[0008] When the operating state indicates that the object to be detected is in a fault state, determining a fault recovery strategy corresponding to the fault state;

[0009] According to the fault recovery strategy, the object to be detected that is in the fault state is repaired.

[0010] On the other hand, the present application also provides a database fault processing device. The device includes:

[0011] An acquisition module, used for acquiring operation data of multi-level objects to be detected in a database through a multi-level state machine, wherein each level of the multi-level state machine corresponds one-to-one to each level of the multi-level objects to be detected, and the state machine of each level is used for acquiring operation data of the objects to be detected at the corresponding level;

[0012] A state determination module, used to determine the operation state of the object to be detected at each level according to the operation data of the multi-level objects to be detected;

[0013] A strategy determination module, used for determining a fault recovery strategy corresponding to the fault state when the operating state indicates that the object to be detected is in a fault state;

[0014] The repair module is used to repair the object to be detected in the fault state according to the fault recovery strategy.

[0015] On the other hand, the present application also provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0016] Acquire the operation data of the multi-level objects to be detected in the database through a multi-level state machine, wherein each level of the multi-level state machine corresponds to each level of the multi-level objects to be detected, and the state machine of each level is used to acquire the operation data of the objects to be detected at the corresponding level;

[0017] Determining the operating status of the objects to be detected at each level according to the operating data of the objects to be detected at the multiple levels;

[0018] When the operating state indicates that the object to be detected is in a fault state, determining a fault recovery strategy corresponding to the fault state;

[0019] According to the fault recovery strategy, the object to be detected that is in the fault state is repaired.

[0020] On the other hand, the present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0021] Acquire the operation data of the multi-level objects to be detected in the database through a multi-level state machine, wherein each level of the multi-level state machine corresponds to each level of the multi-level objects to be detected, and the state machine of each level is used to acquire the operation data of the objects to be detected at the corresponding level;

[0022] Determining the operating status of the objects to be detected at each level according to the operating data of the objects to be detected at the multiple levels;

[0023] When the operating state indicates that the object to be detected is in a fault state, determining a fault recovery strategy corresponding to the fault state;

[0024] According to the fault recovery strategy, the object to be detected that is in the fault state is repaired.

[0025] On the other hand, the present application also provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0026] Acquire the operation data of the multi-level objects to be detected in the database through a multi-level state machine, wherein each level of the multi-level state machine corresponds to each level of the multi-level objects to be detected, and the state machine of each level is used to acquire the operation data of the objects to be detected at the corresponding level;

[0027] Determining the operating status of the objects to be detected at each level according to the operating data of the objects to be detected at the multiple levels;

[0028] When the operating state indicates that the object to be detected is in a fault state, determining a fault recovery strategy corresponding to the fault state;

[0029] According to the fault recovery strategy, the object to be detected that is in the fault state is repaired.

[0030] The above-mentioned database fault processing method, device, computer equipment, storage medium and computer program product obtain the operating data of the multi-level objects to be detected in the database through a multi-level state machine, wherein each level of the multi-level state machine corresponds one-to-one to each level of the multi-level objects to be detected, and the state machine of each level is used to obtain the operating data of the objects to be detected at the corresponding level, so that the state machines can be deployed at multiple levels of the server to obtain the operating data more accurately and quickly. According to the operating data of the multi-level objects to be detected, the operating status of the objects to be detected at each level is determined to determine whether the objects to be detected at each level are in a faulty state or a healthy state. When the operating status indicates that the object to be detected is in a faulty state, the fault recovery strategy corresponding to the faulty state is determined, and according to the fault recovery strategy, the objects to be detected in the faulty state are repaired so that the database is restored to a normal working state, thereby realizing automatic detection and recovery of complete database faults, and effectively improving the security and stability of database operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0032] FIG1 is an application environment diagram of a database fault handling method in one embodiment;

[0033] Figure 2 A flowchart of a method for handling database failures in one embodiment;

[0034] Figure 3 is a flowchart of a method for handling database failures in another embodiment;

[0035] Figure 4 A partial structural diagram of a multi-level state machine in one embodiment;

[0036] Figure 5 is a structural block diagram of a database fault processing device in an embodiment;

[0037] Figure 6 FIG. 4 is a diagram showing the internal structure of a server in one embodiment. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0039] The database fault handling method provided in the embodiment of the present application can be applied to the application environment shown in Figure 1. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 obtains the operating data of the multi-level objects to be detected in the database through a multi-level state machine, wherein each level of the multi-level state machine corresponds to each level of the multi-level objects to be detected, and the state machine of each level is used to obtain the operating data of the objects to be detected at the corresponding level. The server 104 determines the operating state of the objects to be detected at each level according to the operating data of the multi-level objects to be detected. When the operating state indicates that the object to be detected is in a fault state, the server 104 determines the fault recovery strategy corresponding to the fault state. The server 104 repairs the object to be detected in the fault state according to the fault recovery strategy to obtain the repair result. The server 104 feeds back the repair result to the terminal 102.

[0040] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0041] In one embodiment, Figure 2As shown, a database fault processing method is provided, which is described by taking the method applied to the server in FIG1 as an example, and includes the following steps:

[0042] Step 202, obtaining the operating data of the multi-level objects to be detected in the database through a multi-level state machine, wherein each level of the multi-level state machine corresponds one-to-one to each level of the multi-level objects to be detected, and the state machine of each level is used to obtain the operating data of the objects to be detected at the corresponding level.

[0043] The operation data includes at least one of the following: CPU utilization, memory utilization, disk utilization, and network bandwidth.

[0044] The database includes multiple levels of objects to be detected, and the multiple levels of objects to be detected include objects to be detected in at least two levels. The multiple levels of objects to be detected can be clusters, components and nodes, and the clusters, components and nodes belong to different levels, and there is an affiliation relationship between the levels. Each cluster includes at least one component, and each component includes at least one node.

[0045] Each level of the multi-level state machine corresponds to each level of the multi-level object to be detected. For example, the multi-level refers to a set of three levels. The multi-level objects to be detected include clusters, components, and nodes. The multi-level state machine includes a cluster state machine, a component state machine, and a node state machine. The cluster state machine is used to obtain the operation data of the cluster, the component state machine is used to obtain the operation data of the component, and the node state machine is used to obtain the operation data of the node.

[0046] Specifically, the server pre-configures a multi-level state machine for the multi-level objects to be detected in the database. In response to a fault detection trigger event for the database, the server obtains the operation data of the multi-level objects to be detected in the database through the multi-level state machine.

[0047] A fault detection trigger event refers to an event that triggers fault detection, which may be a periodic automatic trigger event or may be triggered by a user.

[0048] In this embodiment, in response to a fault detection trigger event for a database, the server obtains the operation data of multiple levels of objects to be detected in the database within a historical time period through a multi-level state machine.

[0049] Step 204: Determine the operating status of the objects to be detected at each level according to the operating data of the objects to be detected at multiple levels.

[0050] The operating status includes fault status, healthy status and waiting status.

[0051] Specifically, the server determines the operating status of the objects to be detected at each level according to the operating data of the objects to be detected at multiple levels.

[0052] Furthermore, for the objects to be detected at each level, the server determines the operating status of the objects to be detected at the targeted level according to the operating data of the objects to be detected at the targeted level.

[0053] In one of the embodiments, the server may determine the level to which the object to be detected at the targeted level belongs, and determine the operating status of the object to be detected at the targeted level based on the operating data of the object to be detected at the targeted level and the operating data of the objects to be detected at the level to which the object to be detected at the targeted level belongs.

[0054] Step 206: When the running status indicates that the object to be detected is in a fault state, a fault recovery strategy corresponding to the fault state is determined.

[0055] The fault recovery strategy refers to a strategy for eliminating the faults existing in the object to be detected. The fault recovery strategy defines at least one preset recovery action that can repair the faults existing in the object to be detected.

[0056] Specifically, when the running status indicates that the object to be detected is in a fault state, the server determines a fault recovery strategy corresponding to the fault state.

[0057] In this embodiment, when the running status indicates that the object to be detected is in a fault state, the server determines a fault recovery strategy set corresponding to the database, which includes a combination of fault states and fault recovery strategies; and determines the fault recovery strategy corresponding to the fault state from the fault recovery strategy set.

[0058] In this embodiment, when the running status indicates that the object to be detected is in a healthy state, the process returns to the step of obtaining the running data of the multi-level objects to be detected in the database through the multi-level state machine and continues to execute.

[0059] Step 208: repair the object to be detected that is in a fault state according to the fault recovery strategy.

[0060] Specifically, the server executes a preset recovery action specified in the fault recovery strategy to repair the object to be detected that is in a faulty state, so that the object to be detected is adjusted from the faulty state to a healthy state.

[0061] In one embodiment, when the target fault recovery strategy fails to repair the object to be detected in a faulty state, the confidence value of the target fault recovery strategy is reduced.

[0062] In one of the embodiments, when the target fault recovery strategy has repaired the object to be detected that is in a faulty state, the confidence value of the target fault recovery strategy is increased.

[0063] In the above database fault handling method, the operation data of the multi-level objects to be detected in the database are obtained through a multi-level state machine, wherein each level of the multi-level state machine corresponds one-to-one to each level of the multi-level objects to be detected, and the state machine of each level is used to obtain the operation data of the objects to be detected at the corresponding level, so that the state machines can be deployed at multiple levels of the server to obtain the operation data more accurately and quickly. According to the operation data of the multi-level objects to be detected, the operation status of the objects to be detected at each level is determined to determine whether the objects to be detected at each level are in a faulty state or a healthy state. When the operation status indicates that the object to be detected is in a faulty state, the fault recovery strategy corresponding to the fault state is determined, and according to the fault recovery strategy, the object to be detected in the faulty state is repaired, so that the database is restored to a normal working state, thereby realizing automatic detection and recovery of database faults, improving the efficiency of database fault detection and recovery, and effectively improving the security and stability of database operation.

[0064] In one embodiment, when the running state indicates that the object to be detected is in a fault state, determining a fault recovery strategy corresponding to the fault state includes:

[0065] When the running state indicates that the object to be detected is in a fault state, determining multiple candidate fault recovery strategies corresponding to the fault state; determining the confidence values ​​of the multiple candidate fault recovery strategies respectively;

[0066] According to the confidence values ​​of the multiple candidate fault recovery strategies, a target fault recovery strategy corresponding to the fault state is screened out from the multiple candidate fault recovery strategies.

[0067] The confidence value of the fault recovery strategy refers to the score of being able to repair the fault.

[0068] Specifically, when the running state indicates that the object to be detected is in a fault state, the server determines multiple candidate fault recovery strategies corresponding to the fault state from the fault recovery strategy set. The fault strategy set stores multiple preset fault recovery strategies and the confidence value of each fault recovery strategy for at least one fault state.

[0069] The server determines the confidence values ​​of the multiple candidate fault recovery strategies, and based on the confidence values ​​of the multiple candidate fault recovery strategies, selects the candidate fault recovery strategies whose confidence values ​​meet the preset confidence threshold from the multiple candidate fault recovery strategies as the target fault recovery strategies corresponding to the fault state. For example, the candidate fault recovery strategies whose confidence values ​​are greater than the preset confidence threshold are selected as the target recovery strategies.

[0070] In this embodiment, a candidate fault recovery strategy with the highest confidence value may be selected from a plurality of candidate fault recovery strategies as the target recovery strategy.

[0071] In this embodiment, a corresponding fault recovery strategy is set in advance for each fault state. When the running state indicates that the object to be detected is in a fault state, multiple candidate fault recovery strategies corresponding to the fault state are determined, and the confidence values ​​of the multiple candidate fault recovery strategies are determined to determine the probability that the candidate fault recovery strategies can eliminate the fault. According to the confidence values ​​of the multiple candidate fault recovery strategies, the target fault recovery strategy that best matches the fault state is screened out from the multiple candidate fault recovery strategies, so as to accurately eliminate the fault existing in the object to be detected according to the target fault recovery strategy, so that the recovery of the database fault is more targeted.

[0072] In one embodiment, the method further includes: when the target fault recovery strategy has not repaired the object to be detected in a faulty state, reducing the confidence value of the target fault recovery strategy; when the target fault recovery strategy has repaired the object to be detected in a faulty state, increasing the confidence value of the target fault recovery strategy.

[0073] Specifically, the same fault recovery strategy has different confidence values ​​for different fault states. After the server executes the preset recovery action indicated by the target fault recovery strategy for the object to be detected in the fault state, it detects whether the fault of the object to be detected has been repaired. When the target fault recovery strategy fails to repair the object to be detected in the fault state, the confidence value of the target fault recovery strategy for the fault state is reduced.

[0074] When the target fault recovery strategy has repaired the object to be detected that is in a fault state, the confidence value of the target fault recovery strategy for the fault state is increased.

[0075] In this embodiment, when the target fault recovery strategy has not repaired the object to be detected in a faulty state, it means that the target fault recovery strategy has a poor recovery effect on the faulty state, and the confidence value of the target fault recovery strategy for the faulty state is reduced, so that a more effective recovery strategy can be selected for the faulty state in subsequent fault recovery. When the target fault recovery strategy has repaired the object to be detected in a faulty state, it means that the target fault recovery strategy has a good recovery effect on the faulty state, and the confidence value of the target fault recovery strategy is increased, so that a more effective recovery strategy can be selected for this faulty state in subsequent fault recovery, making the fault recovery more accurate.

[0076] In one embodiment, when the target fault recovery strategy fails to repair the object to be detected in a faulty state, reducing the confidence value of the target fault recovery strategy includes:

[0077] When the target fault recovery strategy fails to repair the object to be detected in a faulty state, a penalty value is determined; the confidence value of the target fault recovery strategy is reduced by the penalty value to obtain an updated confidence value of the target fault recovery strategy.

[0078] The penalty value refers to the branch that needs to be lowered when the fault recovery strategy fails to repair a fault state.

[0079] Specifically, when the target fault recovery strategy fails to repair the object to be detected in a fault state, a penalty value of the target fault recovery strategy for the fault state is determined, and the confidence value of the target fault recovery strategy is reduced by the penalty value to obtain an updated confidence value of the target fault recovery strategy.

[0080] In this embodiment, when the target fault recovery strategy fails to repair the object to be detected in the fault state, it indicates that the target fault recovery strategy has a poor recovery effect for the fault state. A penalty value is determined, and the confidence value of the target fault recovery strategy is reduced by the penalty value to reduce the confidence value of the target fault recovery strategy for the fault state, so that a more effective recovery strategy can be selected for the fault state during subsequent fault recovery.

[0081] In one embodiment, when the target fault recovery strategy has repaired the object to be detected in a faulty state, increasing the confidence value of the target fault recovery strategy includes:

[0082] When the target fault recovery strategy has repaired the object to be detected in a faulty state, a reward value is determined; the reward value is added to the confidence value of the target fault recovery strategy to obtain an updated confidence value of the target fault recovery strategy.

[0083] The reward value refers to the branch that needs to be reduced when the fault recovery strategy successfully repairs a fault state.

[0084] Specifically, when the target fault recovery strategy fails to repair the object to be detected in a fault state, the reward value of the target fault recovery strategy for the fault state is determined, and the confidence value of the target fault recovery strategy is reduced by the reward value to obtain an updated confidence value of the target fault recovery strategy.

[0085] In this embodiment, when the target fault recovery strategy fails to repair the object to be detected in the fault state, it means that the target fault recovery strategy has a good recovery effect for the fault state. A reward value is determined, and the confidence value of the target fault recovery strategy is reduced by the reward value to increase the confidence value of the target fault recovery strategy for the fault state, so that the recovery strategy can be given priority for repair when the fault state occurs later, so that the repair of the fault state is more accurate.

[0086] In one embodiment, the multi-level state machine includes a cluster state machine, a component state machine and a node state machine, and the multi-level objects to be detected include a cluster, components under the cluster and nodes under the components.

[0087] The cluster state machine is used to obtain the running data of the cluster, the component state machine is used to obtain the running data of the component, and the node state machine is used to obtain the running data of the node.

[0088] Specifically, the database includes at least one cluster, each cluster includes at least one component, and each component includes at least one node. The server pre-configures a cluster state machine for the database cluster, a component state machine for the database component, and a node state machine for the database node.

[0089] The cluster state machine can monitor the operation of the cluster, the component state machine can monitor the operation of the component, and the node state machine can monitor the operation of the node. Therefore, the server can obtain the operation data of the cluster through the cluster state machine, obtain the operation data of the component through the component state machine, and obtain the operation data of the node through the node state machine.

[0090] In this embodiment, the server can obtain the operation data of the cluster through the cluster state machine, obtain the operation data of the component through the component state machine, and obtain the operation data of the node through the node state machine. According to the operation data of the cluster, component and node, the operation status of the cluster, component and node can be determined, so that the monitoring service can be deployed at the three levels of computer cluster, component and node to obtain the operation data more accurately and quickly, and then locate the fault type. When the operation status indicates that the object to be detected is in a fault state, the fault recovery strategy corresponding to the fault state is determined, so as to automatically respond to the fault recovery strategy, repair the object to be detected in the fault state, and restore the database to a normal working state, so as to achieve fully automated operation and maintenance, avoid the time delay and error cost of manual fault handling, minimize the impact of machine failure, network failure, etc. on the operation operation, ensure the smooth operation of massive data, and greatly improve the security and stability of the database system.

[0091] In one embodiment, a database fault processing method is provided, which is applied to a server and includes:

[0092] Obtain cluster operation data through the cluster state machine, obtain component operation data through the component state machine, and obtain node operation data through the node state machine;

[0093] Determine the operating status of the objects to be detected at each level according to the operating data of the objects to be detected at multiple levels;

[0094] When the running state indicates that the object to be detected is in a fault state, determining multiple candidate fault recovery strategies corresponding to the fault state;

[0095] Determine confidence values ​​of respective ones of a plurality of candidate failure recovery strategies;

[0096] According to the confidence values ​​of the multiple candidate fault recovery strategies, a target fault recovery strategy corresponding to the fault state is screened out from the multiple candidate fault recovery strategies.

[0097] According to the fault recovery strategy, the objects to be detected that are in a faulty state are repaired.

[0098] When the target fault recovery strategy fails to repair the object to be detected in a faulty state, a penalty value is determined;

[0099] The confidence value of the target fault recovery strategy is reduced by the penalty value to obtain an updated confidence value of the target fault recovery strategy.

[0100] When the target fault recovery strategy has repaired the object to be detected in a faulty state, determining a reward value;

[0101] The reward value is added to the confidence value of the target fault recovery strategy to obtain an updated confidence value of the target fault recovery strategy.

[0102] In this embodiment, Figure 3 As shown, a method for handling database failures is provided, the method comprising the following steps:

[0103] 1. State machine design: define the normal operation state and fault state of the distributed database, as well as the transition conditions between states. The state machine adopts a multi-level state machine configuration, which is a multi-level state machine formed by clusters, components, and nodes. Each level of the state machine includes health, fault, waiting and other states. The system uses the agent or the corresponding interface of each component to regularly check the status of each node and component. When the lower-level node state machine enters a fault and the state is stable, it will signal its upper-level state machine, that is, the component state machine. The component state machine will perform the corresponding state flow, and the cluster state machine will also be notified after the component state machine state flow.

[0104] like Figure 4 As shown, it is a schematic diagram of the multi-level state machine in this embodiment. The state machine adopts a multi-level state machine configuration, and the multi-level state machine composed of clusters, components, and nodes, and each level of the state machine includes health, failure, waiting and other states. The system regularly checks the status of each node and component, and updates the status of the node and component according to the detection results. If a node of the segment component loses connection, the node state machine of the segment switches from the RUNNING (running) state to the NODE_OUTOFSERVICE_PEDDING (node ​​offline waiting state). After waiting for a certain period of time, if the health signal is not received, it switches to the NODE_OUTOFSERVICE (node ​​offline state), and sends the signal NODE_OUTOFSERVICE_SIGNAL to the segment component state machine. The component state machine performs the corresponding state transfer and reaches COMPONENT_UNHEALTHY (component failure state), and sends information to the cluster state machine for the corresponding cluster health detection. The final fault state contains detailed information of the fault.

[0105] The database system can be a distributed database system, for example, composed of two clusters, the two clusters are the metadata cluster and the computing cluster. The metadata cluster consists of three components: foundationdb, etcd, and catalog, each with three nodes. The computing cluster consists of master and segment components, the master consists of one node, and the segment consists of two nodes, forming a complete database service.

[0106] 2. Q-Learning algorithm initialization: define all available recovery strategies under different fault conditions, and use prior knowledge and experience to initialize the Q table, that is, under the fault condition, the adoption of this strategy will lead to better results, and the Q value of this strategy can be given a higher initial value. The Q table is the confidence table, and the Q value is the confidence value.

[0107] Specifically, the Q-Learning algorithm initializes a Q table containing the Q values ​​of all possible combinations of "fault state-fault recovery strategy". The initial method is to initialize the Q table based on some prior knowledge or experience. For example, if certain actions usually lead to better results in similar situations, the Q values ​​of these actions can be given higher initial values. When a host failure of a segment is detected, the recovery actions that can be taken include shrink_expand_policy (shrinking the failed node and expanding the node of the same specification), restart_cluster_policy (restarting the cluster), super_shrink_expand_policy (stopping the cluster, shrinking, expanding, and starting the cluster), do_waiting_policy (waiting strategy, no action is performed within the time window, waiting for the metadata cluster it depends on to recover health) and other strategies. Considering that the instance operation and user usage are not affected during the expansion and contraction process, the Q value of shrink_expand_policy is set to 5. The cost of restart is slightly higher, which will temporarily affect user usage. Therefore, the Q value of restart_cluster_policy is set to 4, the Q value of super_shrink_expand_policy is set to 3, and the do_waiting_policy is set to 2.

[0108] The learning rate (α) of the Q table is set to 0.2, the discount factor (γ) is set to 0, and the impact of future states on the current state is not considered for the time being. The calculation formula is:

[0109] Q( s t , a t ) ← Q( s t , a t )+ α [ r t+1 + γ max a Q( s t+1 ,a) - Q( s t , a t ) ]

[0110] in, Represents a combination of a fault state and a fault recovery strategy. Represents the confidence value of the combination. represents the fault status at time t, Represents the fault recovery strategy at time t.

[0111] 3. Fault detection and state transfer: When a node abnormality is detected, the state machine transfers to the corresponding fault state according to preset rules and thresholds.

[0112] 4. Recovery strategy execution: According to the optimal strategy learned by the Q-Learning algorithm, recovery actions are automatically executed, such as node restart, component restart, or node scaling down.

[0113] Specifically, in the state of node fault detection, the Q-Learning algorithm selects a recovery action based on the greedy strategy, that is, the maximum value of the current Q table. The current optimal action is shrink_expand_policy. After executing the action, the system observes the result. If the operation fails (shrinking or expansion fails), the system gives a negative reward of -5; if the operation is successful and the instance is restored to health within the time window, the system gives a positive reward of 5; if the operation is successful, but the instance is not restored to health, the system gives a negative reward of -3; after the same strategy successfully self-heals, if the same fault occurs again within a short period of time (a preset window period, such as 20 minutes), the reward value is reduced by 2.

[0114] 5. Environmental feedback and Q-value update: After executing the recovery action, dynamically update the reward and Q-table based on whether the instance is healthy after executing the recovery action to optimize the recovery strategy. Specifically, Q-value update: Based on the received reward, the Q-Learning algorithm updates the Q-table. Update the segment failure status and execute the Q-value of the action shrink_expand_policy. If the operation is successful, the calculated Q-value remains unchanged. If the operation fails, the Q-value is updated to 3.

[0115] 6. Strategy improvement and state machine update: Integrate the learned optimal strategy into the state machine and update the state transition logic.

[0116] 7. Continuous learning and adaptation: The system continuously learns from new failure recovery attempts and regularly updates its Q table and recovery strategy.

[0117] Through the above steps, the present invention realizes fault detection and recovery by combining the state machine and the Q-Learning algorithm, and significantly improves the automatic recovery capability of the distributed database.

[0118] In this embodiment, the distributed database fault detection and recovery system based on the state machine and Q-Learning algorithm can automatically execute the optimal recovery strategy after detecting a node failure, thereby reducing the failure time. In addition, through continuous learning and adaptation, the system continuously learns from new fault recovery attempts, regularly updates its Q table and recovery strategy to adapt to complex production environments and failure modes, and effectively improves the reliability and stability of the distributed database.

[0119] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0120] Based on the same inventive concept, the embodiment of the present application also provides a database fault handling device for implementing the database fault handling method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more database fault handling device embodiments provided below can refer to the limitations of the database fault handling method above, and will not be repeated here.

[0121] In one embodiment, Figure 5 As shown, a database fault processing device 500 is provided, comprising:

[0122] The acquisition module 502 is used to obtain the operating data of the multi-level objects to be detected in the database through a multi-level state machine, wherein each level of the multi-level state machine corresponds one-to-one to each level of the multi-level objects to be detected, and the state machine of each level is used to obtain the operating data of the objects to be detected at the corresponding level.

[0123] The state determination module 504 is used to determine the operation state of the objects to be detected at each level according to the operation data of the objects to be detected at multiple levels.

[0124] The strategy determination module 506 is used to determine a fault recovery strategy corresponding to the fault state when the running state indicates that the object to be detected is in a fault state.

[0125] The repair module 508 is used to repair the object to be detected that is in a fault state according to the fault recovery strategy.

[0126] In this embodiment, the operation data of the multi-level objects to be detected in the database are obtained through a multi-level state machine, wherein each level of the multi-level state machine corresponds one-to-one to each level of the multi-level objects to be detected, and the state machine of each level is used to obtain the operation data of the objects to be detected at the corresponding level, so that the state machines can be deployed at multiple levels of the server to obtain the operation data more accurately and quickly. According to the operation data of the multi-level objects to be detected, the operation status of the objects to be detected at each level is determined to determine whether the objects to be detected at each level are in a faulty state or a healthy state. When the operation status indicates that the object to be detected is in a faulty state, the fault recovery strategy corresponding to the faulty state is determined, and according to the fault recovery strategy, the objects to be detected in the faulty state are repaired, so that the database is restored to a normal working state, thereby realizing automatic detection and recovery of database faults, improving the efficiency of database fault detection and recovery, and effectively improving the security and stability of database operation.

[0127] In one embodiment, the strategy determination module 506 is further configured to determine a plurality of candidate fault recovery strategies corresponding to the fault state when the running state indicates that the object to be detected is in a fault state;

[0128] Determine the confidence values ​​of the multiple candidate fault recovery strategies respectively; and select the target fault recovery strategy corresponding to the fault state from the multiple candidate fault recovery strategies according to the confidence values ​​of the multiple candidate fault recovery strategies respectively.

[0129] In this embodiment, a corresponding fault recovery strategy is set in advance for each fault state. When the running state indicates that the object to be detected is in a fault state, multiple candidate fault recovery strategies corresponding to the fault state are determined, and the confidence values ​​of the multiple candidate fault recovery strategies are determined to determine the probability that the candidate fault recovery strategies can eliminate the fault. According to the confidence values ​​of the multiple candidate fault recovery strategies, the target fault recovery strategy that best matches the fault state is screened out from the multiple candidate fault recovery strategies, so as to accurately eliminate the fault existing in the object to be detected according to the target fault recovery strategy, so that the recovery of the database fault is more targeted.

[0130] In one embodiment, the strategy determination module 506 is further configured to reduce the confidence value of the target fault recovery strategy when the target fault recovery strategy fails to repair the object to be detected in a faulty state;

[0131] When the target fault recovery strategy has repaired the object to be detected that is in a faulty state, the confidence value of the target fault recovery strategy is increased.

[0132] In this embodiment, when the target fault recovery strategy has not repaired the object to be detected in a faulty state, it means that the target fault recovery strategy has a poor recovery effect on the faulty state, and the confidence value of the target fault recovery strategy for the faulty state is reduced, so that a more effective recovery strategy can be selected for the faulty state in subsequent fault recovery. When the target fault recovery strategy has repaired the object to be detected in a faulty state, it means that the target fault recovery strategy has a good recovery effect on the faulty state, and the confidence value of the target fault recovery strategy is increased, so that a more effective recovery strategy can be selected for this faulty state in subsequent fault recovery, making the fault recovery more accurate.

[0133] In one embodiment, the strategy determination module 506 is further used to determine a penalty value when the target fault recovery strategy fails to repair the object to be detected in a faulty state; reduce the confidence value of the target fault recovery strategy by the penalty value to obtain an updated confidence value of the target fault recovery strategy.

[0134] In this embodiment, when the target fault recovery strategy fails to repair the object to be detected in the fault state, it indicates that the target fault recovery strategy has a poor recovery effect for the fault state. A penalty value is determined, and the confidence value of the target fault recovery strategy is reduced by the penalty value to reduce the confidence value of the target fault recovery strategy for the fault state, so that a more effective recovery strategy can be selected for the fault state during subsequent fault recovery.

[0135] In one embodiment, the strategy determination module 506 is also used to determine a reward value when the target fault recovery strategy has repaired the object to be detected in a faulty state; add the reward value to the confidence value of the target fault recovery strategy to obtain an updated confidence value of the target fault recovery strategy.

[0136] In this embodiment, when the target fault recovery strategy fails to repair the object to be detected in the fault state, it means that the target fault recovery strategy has a good recovery effect for the fault state. A reward value is determined, and the confidence value of the target fault recovery strategy is reduced by the reward value to increase the confidence value of the target fault recovery strategy for the fault state, so that the recovery strategy can be given priority for repair when the fault state occurs later, so that the repair of the fault state is more accurate.

[0137] In one embodiment, the multi-level state machine includes a cluster state machine, a component state machine and a node state machine, and the multi-level objects to be detected include a cluster, components under the cluster and nodes under the components.

[0138] In this embodiment, the server can obtain the operation data of the cluster through the cluster state machine, obtain the operation data of the component through the component state machine, and obtain the operation data of the node through the node state machine. According to the operation data of the cluster, component and node, the operation status of the cluster, component and node can be determined, so that the monitoring service can be deployed at the three levels of computer cluster, component and node to obtain the operation data more accurately and quickly, and then locate the fault type. When the operation status indicates that the object to be detected is in a fault state, the fault recovery strategy corresponding to the fault state is determined, so as to automatically respond to the fault recovery strategy, repair the object to be detected in the fault state, and restore the database to a normal working state, so as to achieve fully automated operation and maintenance, avoid the time delay and error cost of manual fault handling, minimize the impact of machine failure, network failure, etc. on the operation operation, ensure the smooth operation of massive data, and greatly improve the security and stability of the database system.

[0139] Each module in the above-mentioned database fault processing device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0140] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store database fault processing data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a database fault processing method is implemented.

[0141] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0142] The embodiment of the present application further provides a computer-readable storage medium, one or more non-volatile computer-readable storage media containing computer-executable instructions, which, when executed by one or more processors, enable the processors to execute the steps of the database fault handling method.

[0143] The embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute a database fault handling method.

[0144] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0145] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0146] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0147] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A database fault processing method, characterized in that: The method comprises: Acquire the operation data of the multi-level objects to be detected in the database through a multi-level state machine, wherein each level of the multi-level state machine corresponds to each level of the multi-level objects to be detected, and the state machine of each level is used to acquire the operation data of the objects to be detected at the corresponding level; Determining the operating status of the objects to be detected at each level according to the operating data of the objects to be detected at the multiple levels; When the operating state indicates that the object to be detected is in a fault state, determining a fault recovery strategy corresponding to the fault state; According to the fault recovery strategy, the object to be detected that is in the fault state is repaired.

2. The method according to claim 1, characterized in that When the operating state indicates that the object to be detected is in a fault state, determining a fault recovery strategy corresponding to the fault state includes: When the operating state indicates that the object to be detected is in a fault state, determining a plurality of candidate fault recovery strategies corresponding to the fault state; Determining a confidence value for each of the plurality of candidate fault recovery strategies; According to the confidence values ​​of the multiple candidate fault recovery strategies, a target fault recovery strategy corresponding to the fault state is screened out from the multiple candidate fault recovery strategies.

3. The method according to claim 2, characterized in that The method further comprises: When the target fault recovery strategy fails to repair the object to be detected in the fault state, reducing the confidence value of the target fault recovery strategy; When the target fault recovery strategy has repaired the object to be detected in the fault state, the confidence value of the target fault recovery strategy is increased.

4. The method according to claim 3, characterized in that When the target fault recovery strategy fails to repair the object to be detected in the fault state, reducing the confidence value of the target fault recovery strategy includes: When the target fault recovery strategy fails to repair the object to be detected in the fault state, determining a penalty value; The confidence value of the target fault recovery strategy is reduced by the penalty value to obtain an updated confidence value of the target fault recovery strategy.

5. The method according to claim 3, characterized in that: When the target fault recovery strategy has repaired the object to be detected in the fault state, increasing the confidence value of the target fault recovery strategy includes: When the target fault recovery strategy has repaired the object to be detected in the fault state, determining a reward value; The reward value is added to the confidence value of the target fault recovery strategy to obtain an updated confidence value of the target fault recovery strategy.

6. The method according to claim 1, characterized in that The multi-level state machine includes a cluster state machine, a component state machine and a node state machine, and the multi-level objects to be detected include a cluster, components under the cluster and nodes under the components.

7. A database fault processing device, characterized in that: include: An acquisition module, used for acquiring operation data of multi-level objects to be detected in a database through a multi-level state machine, wherein each level of the multi-level state machine corresponds one-to-one to each level of the multi-level objects to be detected, and the state machine of each level is used for acquiring operation data of the objects to be detected at the corresponding level; A state determination module, used to determine the operation state of the object to be detected at each level according to the operation data of the multi-level objects to be detected; A strategy determination module, used for determining a fault recovery strategy corresponding to the fault state when the operating state indicates that the object to be detected is in a fault state; The repair module is used to repair the object to be detected in the fault state according to the fault recovery strategy.

8. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the computer program is executed by the processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images