Fault detection and recovery method, system and device for distributed database and readable storage medium

Abstract

The embodiment of the invention relates to the field of micro-services, and provides a fault detection and recovery method, system and device for a distributed database and a readable storage medium, the method comprises the following steps: sending heartbeat detection data to a data master node at regular time through a cluster management node, the timing period being configurable; if the number of continuous failures reaches a preset threshold value, determining that the data master node fails, and initiating master and standby failure switching; the total fault detection time is calculated based on the number of continuous failures and the timing period. By configuring fixed heartbeat detection times and detection periods, misjudgment possibly caused by uncertain factors such as network delay and jitter between the main node and the standby node of the distributed database is effectively avoided; through active detection and rapid confirmation, rapid fault switching within a few seconds is achieved, the RTO recovery duration is greatly shortened, and existing RPO = 0 and RTOlt are achieved; and on the basis of the 30-second disaster recovery demand, the RTO can reach the disaster recovery demand of several seconds in partial fault scenes.

Description

Technical Field

[0001] The present application relates to the field of computer information technology, and in particular to a distributed database fault detection and recovery method, system, device and readable storage medium. Background Art

[0002] In today's digital age, financial institutions are increasingly dependent on data, and the stability and security of data centers have become the key to the continuity of financial business. In order to cope with various possible disasters and failures, financial data centers must establish a complete disaster recovery system. According to the "Guidelines for Disaster Recovery Construction of Financial Data Centers JR / T0264-2024" and the "Database Government Procurement Requirements Standard-2023", the disaster recovery system is divided into three levels: local disaster recovery, remote disaster recovery, and extreme disaster recovery, aiming to ensure that the data center can quickly restore services and ensure data integrity and business continuity when facing disasters of different scales and natures.

[0003] In-city disaster recovery establishes production centers and disaster recovery centers in the same city to prevent small-scale disasters such as power outages and fires in the city; off-site disaster recovery establishes data centers in different cities to cope with larger regional disasters such as earthquakes and floods; extreme disaster recovery prepares for the worst-case scenario and ensures data security and business continuity in extreme situations.

[0004] On this basis, the construction of the disaster recovery system also involves key performance indicators - recovery time objective (RTO) and data recovery point objective (RPO). These indicators define the maximum allowable time for system service restoration and data recovery in the event of a failure, which is crucial to ensuring the stability of financial services and customer trust.

[0005] However, traditional fault detection and switching mechanisms may not be able to meet the ideal RTO and RPO requirements when facing certain specific fault scenarios, such as server power outages, network outages, or process failures. Therefore, in addition to the general timed heartbeat detection mechanism, an active detection algorithm mechanism needs to be introduced to achieve faster fault detection and recovery. This solution that combines timed heartbeat detection and active detection can significantly reduce the RTO recovery time in specific scenarios. To this end, proposing a more complete and reasonable technical solution for distributed database fault detection and recovery is a technical problem that needs to be solved urgently by technical personnel in this field. Summary of the invention

[0006] The purpose of this application is to target some faults that can be quickly identified: such as server power failure, network disconnection, process failure or HANG dead state, etc., through a timed active detection mechanism, fast fault switching can be achieved within a few seconds, and combined with a general heartbeat detection solution, a more complete and reasonable fault detection and recovery solution can be constructed. In response to the above problems, the technical solutions adopted by this application are as follows:

[0007] In a first aspect, an embodiment of the present application provides a distributed database fault detection and recovery method, which is applicable to general scenarios to avoid misjudgments that may be caused by uncertain factors such as network delays and jitters, and includes the following steps:

[0008] The cluster management node sends heartbeat detection data to the data master node at regular intervals, and the timing period is configurable;

[0009] If the number of consecutive failures reaches the preset threshold, the data master node is considered to be faulty and a master-slave failover is initiated;

[0010] The total fault detection time is calculated based on the number of consecutive failures and the timing period.

[0011] Furthermore, the timing period is set to 8 seconds, and the threshold of consecutive failure times is set to 3.

[0012] Furthermore, DBAgent periodically and proactively checks the process status of the data master node;

[0013] If the normal state of the process is not detected for more than a few cycles, the DBAgent reports it to the cluster management node, which initiates the failover process;

[0014] The timing period and the number of periods are both configurable, and the total fault detection time is better than the general heartbeat detection mechanism.

[0015] Furthermore, the data standby node periodically initiates heartbeat detection to the primary node, and the global parameter count_loss records the number of heartbeat losses;

[0016] If count_loss reaches the preset threshold, the master node is suspected to be faulty and the heartbeat loss event is reported to the cluster management node;

[0017] The cluster management node actively initiates heartbeat detection from all standby machines to the master to determine whether the master node is faulty and initiates master-slave switching;

[0018] The total fault detection time is better than the general heartbeat detection mechanism.

[0019] In a second aspect, the present invention provides a distributed database fault detection and recovery system, which is used in general scenarios to avoid misjudgments that may be caused by uncertain factors such as network delays and jitters. The system includes: a data transmission module, a fault switching module, and a data calculation module, wherein:

[0020] Data transmission module: used for cluster management nodes to periodically send heartbeat detection data to data master nodes, and the timing period is configurable;

[0021] Failure switching module: used to identify the failure of the data master node and initiate the master-slave failure switching when the number of consecutive failures reaches the preset threshold;

[0022] Data calculation module: used to calculate the total fault detection time based on the number of consecutive failures and the timing period.

[0023] Furthermore, the timing period is set to 8 seconds, and the threshold of consecutive failure times is set to 3.

[0024] Furthermore, DBAgent periodically and proactively checks the process status of the data master node;

[0025] If the normal state of the process is not detected for more than a few cycles, the DBAgent reports it to the cluster management node, which initiates the failover process;

[0026] The timing period and the number of periods are both configurable, and the total fault detection time is better than the general heartbeat detection mechanism.

[0027] Furthermore, the data standby node periodically initiates heartbeat detection to the primary node, and the global parameter count_loss records the number of heartbeat losses;

[0028] If count_loss reaches the preset threshold, the master node is suspected to be faulty and the heartbeat loss event is reported to the cluster management node;

[0029] The cluster management node actively initiates heartbeat detection from all standby machines to the master to determine whether the master node is faulty and initiates master-slave switching;

[0030] The total fault detection time is better than the general heartbeat detection mechanism.

[0031] In a third aspect, the present invention provides an electronic device, including a processor and a memory;

[0032] The memory is used to store operation instructions;

[0033] The processor is used to execute the above-mentioned distributed database fault detection and recovery method by calling the operation instruction.

[0034] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for fault detection and recovery of the above-mentioned distributed database is implemented.

[0035] The fault detection and recovery scheme of the distributed database disclosed in the present application sends heartbeat detection data to the data master node periodically through the cluster management node, and the timing period is configurable; if the number of consecutive failures reaches the preset threshold, the data master node is deemed to be faulty, and the master-slave fault switching is initiated; the total fault detection time is calculated based on the number of consecutive failures and the timing period. The beneficial effect brought by the technical solution provided by the present application is that by configuring a fixed number of heartbeat detections and detection cycles, the misjudgment that may be caused by uncertain factors such as network delays and jitters between the master and standby nodes of the distributed database can be effectively avoided, and it has universal applicability. Through active detection and rapid confirmation, rapid fault switching can be achieved within a few seconds, thereby greatly reducing the RTO recovery time, and achieving the disaster recovery requirement of RTO reaching a few seconds in some fault scenarios based on the original RPO=0,RTO<30 seconds disaster recovery requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in describing the embodiments of the present application.

[0037] Figure 1 A schematic diagram of a flow chart of a distributed database fault detection and recovery method provided in an embodiment of the present application;

[0038] Figure 2 A flowchart of a distributed database fault detection and recovery method provided in an embodiment of the present application;

[0039] Figure 3 A flowchart of a distributed database fault detection and recovery method provided in an embodiment of the present application;

[0040] Figure 4 A flowchart of a distributed database fault detection and recovery method provided in an embodiment of the present application;

[0041] Figure 5 A flowchart of a distributed database fault detection and recovery method provided in an embodiment of the present application;

[0042] Figure 6 A schematic diagram of the structure of a distributed database fault detection and recovery system provided in an embodiment of the present application;

[0043] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0045] It will be understood by those skilled in the art that, unless otherwise stated, the singular forms "a", "an" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0046] In order to make the objectives, technical solutions and advantages of the present application more clear, the specific implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0047] The core of the present invention is to provide a distributed database fault detection and recovery method. Figure 1 The figure is a flow chart of the fault detection and recovery method of the distributed database provided in this embodiment. In order to better understand the method, Figure 1 The technical solution of this embodiment is described in detail:

[0048] The method of this embodiment is applicable to general scenarios to avoid misjudgment caused by uncertain factors such as network delay and jitter, and includes the following steps:

[0049] Step S101: The cluster management node periodically sends heartbeat detection data to the data master node, and the timing period is configurable;

[0050] Step S102: If the number of consecutive failures reaches a preset threshold, the data master node is deemed to be faulty, and a master-slave fault switch is initiated;

[0051] Step S103: The total fault detection time is calculated based on the number of consecutive failures and the timing period.

[0052] As a preferred embodiment of this embodiment, the timing period is set to 8 seconds, and the threshold of consecutive failure times is set to 3.

[0053] As a preferred embodiment of this embodiment, it further includes:

[0054] DBAgent regularly and proactively checks the process status of the data master node;

[0055] If the normal state of the process is not detected for more than a few cycles, the DBAgent reports it to the cluster management node, which initiates the failover process;

[0056] The timing period and the number of periods are both configurable, and the total fault detection time is better than the general heartbeat detection mechanism.

[0057] As a preferred embodiment of this embodiment, it further includes:

[0058] The data standby node periodically initiates heartbeat detection to the primary node, and the global parameter count_loss records the number of heartbeat losses;

[0059] If count_loss reaches the preset threshold, the master node is suspected to be faulty and the heartbeat loss event is reported to the cluster management node;

[0060] The cluster management node actively initiates heartbeat detection from all standby machines to the master to determine whether the master node is faulty and initiates master-slave switching;

[0061] The total fault detection time is better than the general heartbeat detection mechanism.

[0062] The following will introduce the common heartbeat detection mechanism of distributed databases and the specific implementation process of the active detection algorithm mechanism optimized for some scenarios.

[0063] 1. General heartbeat detection mechanism, see the corresponding flow chart Figure 2 shown.

[0064] 1. The cluster management node periodically detects the data master node through heartbeat detection. The timing period can be configured, such as 8 seconds.

[0065] 2. If the number of consecutive heartbeat detection failures reaches a certain threshold, the data master node is considered to be faulty and a master-slave failover is initiated immediately; the failure threshold is configurable, such as 3.

[0066] 3. The total fault detection time is: 3*8=24 seconds

[0067] 4. This mechanism can be applied to general scenarios and can avoid misjudgments caused by uncertain factors such as network delays and jitters between the master and standby nodes of distributed databases.

[0068] 2. Active detection algorithm mechanism, see the corresponding flowchart Figure 3 shown.

[0069] Based on the general heartbeat detection solution, a timed active detection mechanism is added. The detailed mechanism is as follows:

[0070] 1. Regularly detect the status of the main process:

[0071] 1) DBAgent regularly and proactively checks the process status ( / etc / pid / stat) of the corresponding data master node. If the normal state of the process is not detected for more than a few cycles, it will be reported to the cluster management node immediately, and the cluster management node will immediately initiate the failover process;

[0072] 2) The timing period can be configured: for example, set it to 1 second and the number of cycles to 2 times.

[0073] 3) The total fault detection time is 2 seconds, which is much better than the general heartbeat detection mechanism.

[0074] 4) Mainly for specific scenarios such as process failure or HANG dead state.

[0075] 5) Since DBAgnet periodically detects the main process of the data node on the same server, such as every 1 second, the performance impact can be ignored.

[0076] 2. Event-triggered liveness detection. For the corresponding flowchart, see Figure 4 As shown;

[0077] 1) The data standby node periodically initiates heartbeat detection to the master node. The timing period is configurable, such as 1 second. The global parameter count_loss is initially 0. If a heartbeat is lost, count_loss++.

[0078] 2) If count_loss reaches a certain threshold (such as 2), it is suspected that the master node may fail, and the heartbeat loss event is immediately reported to the cluster management node.

[0079] 3) The cluster management node actively initiates heartbeat detection from all backup machines to the master machine.

[0080] 4) If all heartbeat detections from the slave database to the master database fail, the master node is considered to be faulty, and a master-slave switch is initiated immediately, while count_loss is reset to zero; otherwise, the process returns to step 1.

[0081] 5) The total fault detection time is 2 seconds, which is much better than the general heartbeat detection mechanism.

[0082] 6) Mainly for specific scenarios such as power outage and network disconnection.

[0083] 7) The standby database periodically pings the primary and standby databases, for example, every 1 second. The performance impact can be ignored.

[0084] In summary, for some faults that can be quickly identified, such as server power failure, network disconnection, process failure or HANG dead state, the timing active detection mechanism can achieve fast fault detection and switching within a few seconds of RTO. Combined with the general heartbeat detection solution, a more complete and reasonable fault detection and recovery solution can be constructed. The complete solution mechanism can be found in the process Figure 5 shown.

[0085] In an embodiment of the present application, the heartbeat detection data is periodically sent to the data master node through the cluster management node, and the timing period is configurable; if the number of consecutive failures reaches a preset threshold, the data master node is deemed to be faulty, and a master-slave fault switch is initiated; the total fault detection time is calculated based on the number of consecutive failures and the timing period. The beneficial effect of the technical solution provided by the present application is that by configuring a fixed number of heartbeat detections and detection cycles, the misjudgment that may be caused by uncertain factors such as network delays and jitters between the master and standby nodes of the distributed database can be effectively avoided, and it has universal applicability. Through active detection and rapid confirmation, rapid fault switching can be achieved within a few seconds, thereby greatly reducing the RTO recovery time, and achieving the disaster recovery requirement of RTO reaching a few seconds in partial fault scenarios based on the original RPO=0, RTO<30 seconds disaster recovery requirements.

[0086] based on Figure 1 The embodiment shown provides a distributed database fault detection and recovery method, Figure 6 A distributed database fault detection and recovery system provided by an embodiment of the present application is shown. Figure 6 As shown, the system is used in general scenarios to avoid misjudgments that may be caused by uncertain factors such as network delay and jitter. The system can mainly include: 201 data transmission module, 202 fault switching module, 203 data calculation module, wherein,

[0087] 201 Data transmission module: used for the cluster management node to periodically send heartbeat detection data to the data master node, and the timing period is configurable;

[0088] 202 Failure switching module: used to identify the data master node failure and initiate the master-slave failure switching when the number of consecutive failures reaches a preset threshold;

[0089] 203 data calculation module: used for calculating the total fault detection time based on the number of consecutive failures and the timing period.

[0090] As a preferred embodiment of this embodiment, the timing period is set to 8 seconds, and the threshold of consecutive failure times is set to 3.

[0091] As a preferred embodiment of this embodiment, it further includes:

[0092] DBAgent regularly and proactively checks the process status of the data master node;

[0093] If the normal state of the process is not detected for more than a few cycles, the DBAgent reports it to the cluster management node, which initiates the failover process;

[0094] The timing period and the number of periods are both configurable, and the total fault detection time is better than the general heartbeat detection mechanism.

[0095] As a preferred embodiment of this embodiment, it further includes:

[0096] The data standby node periodically initiates heartbeat detection to the primary node, and the global parameter count_loss records the number of heartbeat losses;

[0097] If count_loss reaches the preset threshold, the master node is suspected to be faulty and the heartbeat loss event is reported to the cluster management node;

[0098] The cluster management node actively initiates heartbeat detection from all standby machines to the master to determine whether the master node is faulty and initiates master-slave switching;

[0099] The total fault detection time is better than the general heartbeat detection mechanism.

[0100] In an embodiment of the present application, the heartbeat detection data is periodically sent to the data master node through the cluster management node, and the timing period is configurable; if the number of consecutive failures reaches a preset threshold, the data master node is deemed to be faulty, and a master-slave fault switch is initiated; the total fault detection time is calculated based on the number of consecutive failures and the timing period. The beneficial effect of the technical solution provided by the present application is that by configuring a fixed number of heartbeat detections and detection cycles, the misjudgment that may be caused by uncertain factors such as network delays and jitters between the master and standby nodes of the distributed database can be effectively avoided, and it has universal applicability. Through active detection and rapid confirmation, rapid fault switching can be achieved within a few seconds, thereby greatly reducing the RTO recovery time, and achieving the disaster recovery requirement of RTO reaching a few seconds in partial fault scenarios based on the original RPO=0, RTO<30 seconds disaster recovery requirements.

[0101] It can be understood that the above modules of the distributed database fault detection and recovery system in this embodiment have the function of implementing Figure 1 The functions of the corresponding steps of the method in the embodiment shown in . The function can be implemented by hardware, or by hardware executing the corresponding software. The hardware or software includes one or more modules corresponding to the above functions. The above modules can be software and / or hardware, and the above modules can be implemented separately or integrated with multiple modules. For the functional description of the above modules, please refer to Figure 1The corresponding description of the method in the embodiment shown in will not be repeated here.

[0102] An embodiment of the present application provides an electronic device, including a processor and a memory;

[0103] A memory, used for storing operation instructions;

[0104] The processor is used to execute the distributed database fault detection and recovery method provided in any embodiment of the present application by calling an operation instruction.

[0105] As an example, Figure 7 A schematic diagram of the structure of an electronic device applicable to the embodiment of the present application is shown. Figure 7 As shown, the electronic device 2000 includes: a processor 2001 and a memory 2003. The processor 2001 and the memory 2003 are connected, such as through a bus 2002. Optionally, the electronic device 2000 may also include a transceiver 2004. It should be noted that in actual applications, the transceiver 2004 is not limited to one, and the structure of the electronic device 2000 does not constitute a limitation on the embodiments of the present application.

[0106] The processor 2001 is applied in the embodiment of the present application to implement the method shown in the above method embodiment. The transceiver 2004 may include a receiver and a transmitter. The transceiver 2004 is applied in the embodiment of the present application to implement the function of the electronic device of the embodiment of the present application communicating with other devices when executed.

[0107] Processor 2001 may be a CPU (Central Processing Unit), a general purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 2001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0108] The bus 2002 may include a path to transmit information between the above components. The bus 2002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 2002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0109] The memory 2003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0110] Optionally, the memory 2003 is used to store application code for executing the solution of the present application, and the execution is controlled by the processor 2001. The processor 2001 is used to execute the application code stored in the memory 2003 to implement the distributed database fault detection and recovery method provided in any embodiment of the present application.

[0111] The electronic device provided in the embodiment of the present application is applicable to any embodiment of the above method and will not be described in detail here.

[0112] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the distributed database fault detection and recovery method shown in the above method embodiment is implemented.

[0113] The computer-readable storage medium provided in the embodiments of the present application is applicable to any embodiment of the above method and will not be described in detail here.

[0114] The fault detection and recovery scheme for a distributed database provided in an embodiment of the present application periodically sends heartbeat detection data to the data master node through the cluster management node, and the timing period is configurable; if the number of consecutive failures reaches a preset threshold, the data master node is deemed to be faulty, and a master-slave fault switch is initiated; the total fault detection time is calculated based on the number of consecutive failures and the timing period. The beneficial effect of the technical solution provided by the present application is that by configuring a fixed number of heartbeat detections and detection cycles, the misjudgment that may be caused by uncertain factors such as network delays and jitters between the master and standby nodes of the distributed database can be effectively avoided, and it has universal applicability. Through active detection and rapid confirmation, rapid fault switching can be achieved within a few seconds, thereby greatly reducing the RTO recovery time, and achieving the disaster recovery requirement of RTO reaching a few seconds in some fault scenarios based on the original RPO=0, RTO<30 seconds disaster recovery requirements.

[0115] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the specific system can be divided into different functional modules to complete all or part of the functions described above.

[0116] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0117] In several embodiments covered by the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, or it can be an electrical, mechanical or other form of connection.

[0118] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0119] In addition, each functional module in each embodiment of the present application can be implemented in the form of hardware or in the form of software functional modules. If these functional modules are implemented in the form of software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application is essentially or part of the prior art that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a computer program product, which includes one or more computer instructions. When loading and executing computer program instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer program product is stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, a computer instruction can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a solid state disk (SSD)).

[0120] The above contents are merely examples and explanations of the structure of the present invention. The technicians in this technical field may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the structure of the invention or exceed the scope defined by the claims, they should all fall within the protection scope of the present invention.

Claims

1. A distributed database fault detection and recovery method, characterized in that: The method is applicable to general scenarios to avoid misjudgment caused by uncertain factors such as network delay and jitter, and includes the following steps: The cluster management node sends heartbeat detection data to the data master node at regular intervals, and the timing period is configurable; If the number of consecutive failures reaches the preset threshold, the data master node is considered to be faulty and a master-slave failover is initiated; The total fault detection time is calculated based on the number of consecutive failures and the timing period.

2. The distributed database fault detection and recovery method according to claim 1, characterized in that: The timing period is set to 8 seconds, and the consecutive failure threshold is set to 3.

3. The distributed database fault detection and recovery method according to claim 1, characterized in that: Further including: DBAgent regularly and proactively checks the process status of the data master node; If the normal state of the process is not detected for more than a few cycles, the DBAgent reports it to the cluster management node, which initiates the failover process; The timing period and the number of periods are both configurable, and the total fault detection time is better than the general heartbeat detection mechanism.

4. The distributed database fault detection and recovery method according to claim 1, characterized in that: Further including: The data standby node periodically initiates heartbeat detection to the primary node, and the global parameter count_loss records the number of heartbeat losses; If count_loss reaches the preset threshold, the master node is suspected to be faulty and the heartbeat loss event is reported to the cluster management node; The cluster management node actively initiates heartbeat detection from all standby machines to the master to determine whether the master node is faulty and initiates master-slave switching; The total fault detection time is better than the general heartbeat detection mechanism.

5. A distributed database fault detection and recovery system, characterized in that: The system is used in general scenarios to avoid misjudgments caused by uncertain factors such as network delay and jitter. The system includes: a data transmission module, a fault switching module, and a data calculation module, wherein: Data transmission module: used for cluster management nodes to periodically send heartbeat detection data to data master nodes, and the timing period is configurable; Failure switching module: used to identify the failure of the data master node and initiate the master-slave failure switching when the number of consecutive failures reaches the preset threshold; Data calculation module: used to calculate the total fault detection time based on the number of consecutive failures and the timing period.

6. The distributed database fault detection and recovery system according to claim 5, characterized in that: The timing period is set to 8 seconds, and the consecutive failure threshold is set to 3.

7. The distributed database fault detection and recovery system according to claim 5, characterized in that: Further including: DBAgent regularly and proactively checks the process status of the data master node; If the normal state of the process is not detected for more than a few cycles, the DBAgent reports it to the cluster management node, which initiates the failover process; The timing period and the number of periods are both configurable, and the total fault detection time is better than the general heartbeat detection mechanism.

8. The distributed database fault detection and recovery system according to claim 5, characterized in that: The data standby node periodically initiates heartbeat detection to the primary node, and the global parameter count_loss records the number of heartbeat losses; If count_loss reaches the preset threshold, the master node is suspected to be faulty and the heartbeat loss event is reported to the cluster management node; The cluster management node actively initiates heartbeat detection from all standby machines to the master to determine whether the master node is faulty and initiates master-slave switching; The total fault detection time is better than the general heartbeat detection mechanism.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store operation instructions; The processor is used to execute the method according to any one of claims 1 to 4 by calling the operation instruction.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images