Distributed memory database intelligent fault processing method and system
By introducing fault detection and monitoring modules into distributed memory databases, combining detection and monitoring programs to automatically execute fault handling programs, the problem of extended fault handling time in the existing technology is solved, and the reliability and availability of the system are improved.
Patent Information
- Application Number
- CN202410965312.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-11-15
AI Technical Summary
The existing distributed in-memory database cannot automatically detect and handle failures when node services are abnormal or performance degraded, resulting in extended fault handling time and affecting the reliability and availability of the system.
By introducing fault detection modules, monitoring modules and fault processing modules into distributed memory databases, combining detection programs and monitoring programs, the fault types are detected in real time and the corresponding fault processing programs are automatically executed, including main and backup switching, resource adjustment and other measures.
It realizes timely detection and processing of distributed memory database faults, improves the reliability and availability of the system, reduces operation and maintenance costs, and avoids the impact of amplified failures.
Smart Images

Figure CN118964122A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a distributed memory database intelligent fault processing method and system. Background Art
[0002] The current high-availability fault takeover of distributed memory databases includes scenarios of network interruption, host failure, and abnormal node service failure. These scenarios have a common feature that the node service does not exist and the failure has occurred. If the node service is available, but the program is dead or the performance of a single host is degraded, or the thread pool, number of connections, memory resources, pre-parsing and other resources are full, the program is normal but the business access or response request is slow. In this case, the program cannot detect the abnormality and can only be resolved through manual intervention.
[0003] However, manual intervention also means that the overall problem-solving time is relatively long. Personnel need to log in to the environment, locate problems, and solve problems in a timely manner. The above operations will prolong the problem-solving time and affect the timeliness of fault handling. Summary of the invention
[0004] The purpose of the present invention is to provide an efficient distributed memory database intelligent fault handling method and system.
[0005] In order to solve the above technical problems, the present invention provides a distributed memory database intelligent fault handling method, comprising the following steps:
[0006] Perform fault detection on the distributed memory database and obtain the fault type;
[0007] Get the fault handling procedure according to the fault type;
[0008] Execute the fault handling program to process the distributed memory database.
[0009] Preferably, performing fault detection on the distributed memory database to obtain the fault type specifically includes the following steps:
[0010] Perform live detection on the distributed memory database to obtain the live detection result;
[0011] Monitor the distributed memory database and obtain the monitoring results;
[0012] According to the detection and monitoring results, the fault type is obtained.
[0013] Preferably, performing live detection on the distributed memory database to obtain live detection results specifically includes the following steps:
[0014] Create a full node table in the distributed memory database;
[0015] The distributed memory database is checked once every minute and the full node table is updated;
[0016] According to the full node table, the live detection result is obtained.
[0017] Preferably, the full node table includes several serial numbers and corresponding operation times.
[0018] Preferably, obtaining the liveness detection result according to the full node table specifically includes the following steps:
[0019] According to the sequence number of that time, obtain the recording time of the sequence number of that time;
[0020] According to the current sequence number recording time and the current system time, the current judgment result is obtained;
[0021] Compare the last sequence number in the full node table to get the last sequence number record time;
[0022] According to the previous sequence number recording time and the corresponding previous operation time, the previous judgment result is obtained;
[0023] The detection result is obtained based on the current judgment result and the previous judgment result.
[0024] Preferably, the distributed memory database is monitored to obtain monitoring results;
[0025] Get network performance metrics in distributed in-memory databases;
[0026] The monitoring results are obtained based on the network performance indicators and the predicted indicator thresholds.
[0027] Preferably, the network performance indicators include network latency, CPU usage, disk usage, memory usage, number of online connections, continuous lock occupancy, thread pool, pre-parsing, intra-cluster synchronization backlog and TPS / QPS value.
[0028] Preferably, according to the fault type, a fault handling procedure is obtained, which specifically includes the following steps:
[0029] Extract features of fault types and obtain key values;
[0030] According to the key value, the fault handling procedure is obtained from the knowledge base.
[0031] The present invention also provides a distributed memory database intelligent fault processing system, comprising:
[0032] A fault detection module is used to detect faults in the distributed memory database and obtain the fault type;
[0033] A program acquisition module is used to obtain a fault handling program according to the fault type;
[0034] The processing module is used to execute the fault handling program to process the distributed memory database.
[0035] Compared with the prior art, the present invention has the following beneficial effects:
[0036] The present invention can realize more timely fault detection and processing, and improve the reliability of the system. By combining with the database monitoring tool, the status and performance indicators of the database can be monitored in real time. At the same time, in conjunction with the detection program, it can detect node deadlock, unreasonable configuration parameters, and other failures that may cause the database to be unable to provide response or slow performance. Once a problem is found, the system can automatically take intervention measures, such as triggering the master-slave switch or forcibly releasing the lock, adjusting the system parameter configuration, etc. In this way, the problem can be solved at the budding stage of the problem to prevent the fault from further expanding its impact.
[0037] The present invention realizes active discovery and resolution of problems in distributed memory databases through cooperation with database monitoring and host detection, thereby improving the availability and stability of the database and reducing operation and maintenance costs. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.
[0039] Figure 1 It is a flow chart of a distributed memory database intelligent fault handling method of the present invention. DETAILED DESCRIPTION
[0040] Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present invention, so the present invention is not limited to the specific implementation disclosed below.
[0041] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.
[0042] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0043] The present invention is further described in detail below in conjunction with the accompanying drawings:
[0044] The present invention provides a distributed memory database intelligent fault processing method, comprising the following steps:
[0045] Perform fault detection on the distributed memory database and obtain the fault type;
[0046] Get the fault handling procedure according to the fault type;
[0047] Execute the fault handling program to process the distributed memory database.
[0048] Preferably, performing fault detection on the distributed memory database to obtain the fault type specifically includes the following steps:
[0049] Perform live detection on the distributed memory database to obtain the live detection result;
[0050] Monitor the distributed memory database and obtain the monitoring results;
[0051] According to the detection and monitoring results, the fault type is obtained.
[0052] Preferably, performing live detection on the distributed memory database to obtain live detection results specifically includes the following steps:
[0053] Create a full node table in the distributed memory database;
[0054] The distributed memory database is checked once every minute and the full node table is updated;
[0055] According to the full node table, the live detection result is obtained.
[0056] Preferably, the full node table includes several serial numbers and corresponding operation times.
[0057] Preferably, obtaining the liveness detection result according to the full node table specifically includes the following steps:
[0058] According to the sequence number of that time, obtain the recording time of the sequence number of that time;
[0059] According to the current sequence number recording time and the current system time, the current judgment result is obtained;
[0060] Compare the last sequence number in the full node table to get the last sequence number record time;
[0061] According to the previous sequence number recording time and the corresponding previous operation time, the previous judgment result is obtained;
[0062] The detection result is obtained based on the current judgment result and the previous judgment result.
[0063] Preferably, the distributed memory database is monitored to obtain monitoring results;
[0064] Get network performance metrics in distributed in-memory databases;
[0065] The monitoring results are obtained based on the network performance indicators and the predicted indicator thresholds.
[0066] Preferably, the network performance indicators include network latency, CPU usage, disk usage, memory usage, number of online connections, continuous lock occupancy, thread pool, pre-parsing, intra-cluster synchronization backlog and TPS / QPS value.
[0067] Preferably, according to the fault type, a fault handling procedure is obtained, which specifically includes the following steps:
[0068] Extract features of fault types and obtain key values;
[0069] According to the key value, the fault handling procedure is obtained from the knowledge base.
[0070] The present invention also provides a distributed memory database intelligent fault processing system, comprising:
[0071] A fault detection module is used to detect faults in the distributed memory database and obtain the fault type;
[0072] A program acquisition module is used to obtain a fault handling program according to the fault type;
[0073] The processing module is used to execute the fault handling program to process the distributed memory database.
[0074] The high availability mechanism of the distributed memory database is usually triggered after a program exception or failure occurs to ensure the stability and availability of the system. The present invention proactively discovers problems and automatically intervenes to solve them in the early stages of the problem by cooperating with the database monitoring and detection program.
[0075] In order to better illustrate the technical effect of the present invention, the present invention provides the following specific embodiments to illustrate the above technical process:
[0076] Embodiment 1: A distributed memory database intelligent fault handling system. Figure 1 As shown in the figure, it consists of four parts: detection program, monitoring program, knowledge base and fault intervention program.
[0077] 1. Exploration program: Create a full-node table in the distributed memory database. The table has two fields. The first field is the sequence number (id) and the second field is the operation time (create_time).
[0078] The id field is (number 6 digits): the record is 1-1440, the result is updated every minute, based on the previous result + 1, and it will be reset at 1440. If the updated result value does not match the current system time, it will enter the inspection mechanism. The fault handling mechanism is initiated and the system time is updated again. For example, if the current time is 3 hours and 50 minutes, the value is 230.
[0079] create_time time field (timestamp): records the current operation time
[0080] The system checks once every minute to update the values of the id and create_time fields. Before each update, it checks the result of the last execution. If the result of the last check does not match the value after the time field value is converted, it will be forced to synchronize again with the system time. If both results do not match the current time, the problem node will be synchronized to the knowledge base, which will judge the problem and execute the fault intervention procedure.
[0081] 2. Monitoring program: Monitors indicators such as distributed memory database network latency, CPU usage, disk usage, memory usage, number of online connections, continuous lock occupancy, thread pool, pre-analysis, cluster synchronization backlog, TPS / QPS value, etc. The monitoring program is scheduled every 10 minutes. When an abnormality is found in the corresponding indicator, the abnormal value will be sent to the knowledge base. The knowledge base judges the sent indicator, generates the corresponding fault intervention program and executes it.
[0082] 3. Knowledge base: Analyze the feedback results of the detection program and the monitoring program. In the knowledge base, each fault feedback result has a corresponding problem handling template. When the knowledge base receives the problem indicator feedback from the monitoring program or the detection program, the knowledge base will find the corresponding fault handling method based on the key value, first check whether the fault matches the current fault handling template, and if so, generate the corresponding fault intervention program based on the template.
[0083] For example, for the live detection program, based on the abnormal nodes reported by the fault live detection program, first check whether the reported nodes cannot provide read and write services. If the requirements are met, the corresponding node suicide command is generated to perform the active / standby switch. When the active / standby switch is completed, the abnormal node is pulled up again. The knowledge base will generate corresponding steps and executable commands for the fault intervention program.
[0084] For faults reported by the monitoring program, such as pre-parsing, memory usage, thread pool and other resource issues, the nodes reported by the alarm are checked first, and then the corresponding expansion commands are generated. For indicators such as TPS / QPS and continuous lock occupancy, the knowledge base will make judgments. If the TPS / QPS value of a node is too different from that of other nodes, the last executed SQL statement will be checked. If there is no problem with the SQL statement check, the master-slave switch-related commands will be generated according to the template and given to the fault intervention program.
[0085] 4. Fault intervention: The steps and commands provided by the knowledge base are integrated into executable programs that can be automatically executed or connected to the customer's management and control platform. After the customer's authorization, the program is scheduled and executed through the management and control platform. After the program is executed, it is synchronized to the log, and the corresponding monitoring program or detection program will be re-executed to record the latest inspection results in the log.
[0086] The present invention integrates four programs: the detection program, the monitoring program, the knowledge base, and the fault intervention program. The knowledge base generates the processing and inspection means of the corresponding problems, and the fault intervention program performs scheduling. It can reduce manual intervention, solve problems intelligently, improve the availability and stability of the database, and reduce operation and maintenance costs.
[0087] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules, modules or units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units, modules or components can be combined or integrated into another device, or some features can be ignored or not executed.
[0088] The units may or may not be physically separated, and the components displayed as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0089] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0090] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part, and / or installed from a removable medium. When the computer program is executed by the central processing unit (CPU), the above-mentioned functions defined in the method of the present invention are executed. It should be noted that the above-mentioned computer-readable medium of the present invention can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of an electrical, magnetic, optical, electromagnetic, infrared segment, or semiconductor, or any combination of the above.
[0091] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0092] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
Claims
1. A distributed memory database intelligent fault handling method, characterized in that: The following steps are involved: Perform fault detection on the distributed memory database and obtain the fault type; Get the fault handling procedure according to the fault type; Execute the fault handling program to process the distributed memory database.
2. A distributed memory database intelligent fault handling method according to claim 1, characterized in that: Perform fault detection on the distributed memory database to obtain the fault type, which specifically includes the following steps: Perform live detection on the distributed memory database to obtain the live detection result; Monitor the distributed memory database and obtain the monitoring results; According to the detection and monitoring results, the fault type is obtained.
3. A distributed memory database intelligent fault handling method according to claim 2, characterized in that: Perform live detection on the distributed memory database to obtain live detection results, which specifically includes the following steps: Create a full node table in the distributed memory database; The distributed memory database is checked once every minute and the full node table is updated; According to the full node table, the live detection result is obtained.
4. The distributed memory database intelligent fault handling method according to claim 3, characterized in that: The full node table includes several serial numbers and corresponding operation times.
5. A distributed memory database intelligent fault handling method according to claim 4, characterized in that: According to the full node table, the live detection result is obtained, which specifically includes the following steps: According to the sequence number of that time, obtain the recording time of the sequence number of that time; According to the current sequence number recording time and the current system time, the current judgment result is obtained; Compare the last sequence number in the full node table to get the last sequence number record time; According to the previous sequence number recording time and the corresponding previous operation time, the previous judgment result is obtained; The detection result is obtained based on the current judgment result and the previous judgment result.
6. A distributed memory database intelligent fault handling method according to claim 5, characterized in that: Monitor the distributed memory database and obtain the monitoring results; Get network performance metrics in distributed in-memory databases; The monitoring results are obtained based on the network performance indicators and the predicted indicator thresholds.
7. The distributed memory database intelligent fault handling method according to claim 6, characterized in that: The network performance indicators include network latency, CPU usage, disk usage, memory usage, number of online connections, continuous lock occupancy, thread pool, pre-parsing, intra-cluster synchronization backlog and TPS / QPS value.
8. The distributed memory database intelligent fault handling method according to claim 7, characterized in that: According to the fault type, obtain the fault handling procedure, which includes the following steps: Extract features of fault types and obtain key values; According to the key value, the fault handling procedure is obtained from the knowledge base.
9. A distributed memory database intelligent fault handling system, used to implement the distributed memory database intelligent fault handling method according to any one of claims 1 to 8, characterized in that: include: A fault detection module is used to detect faults in the distributed memory database and obtain the fault type; A program acquisition module is used to obtain a fault handling program according to the fault type; The processing module is used to execute the fault handling program to process the distributed memory database.
