Cluster service processing method, server, and system

By introducing IO flow detection services into the cluster business system, combined with the heartbeat detection results, the problem that heartbeat network detection cannot detect RAID card failure is solved, and service switching is realized in the case of soft failure of RAID card, improving system reliability and service continuity is achieved. sex.

WO2024239569A1PCT designated stage expired Publication Date: 2024-11-28XFUSION DIGITAL TECH CO LTD

Patent Information

Application Number
PCT/CN2023/134453
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-19
Filing Date
2023-11-27
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

In the prior art, high-availability clusters cannot detect RAID card failure through heartbeat network detection, resulting in the inability to switch from the failed node to the normal node, affecting the continuity of the database service.

Method used

IO flow detection service is introduced, each node detects local IO flow separately, performs peer-side IO flow detection when the detection result is abnormal, determines the cause of the failure based on the heartbeat detection result, and conducts service switching.

Benefits of technology

It improves the reliability of the cluster business system, ensures that the service can be switched to normal nodes in a timely manner when the RAID card fails, and avoids system fake death and data reading and writing problems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2023134453_28112024_PF_FP_ABST
    Figure CN2023134453_28112024_PF_FP_ABST
Patent Text Reader

Abstract

The present application provides a cluster service processing method, a server, and a system. The method is applied to a cluster service system, and the cluster service system comprises a first node and a second node. The method comprises: when a heartbeat network of a first node and a heartbeat network of a second node are normal, acquiring a detection result of local input / output (IO) stream detection; when the detection result of the local IO stream detection is anomalous, acquiring a detection result of opposite-end IO stream detection; and determining a fault reason on the basis of the detection result of the opposite-end IO stream detection and the detection result of the local IO stream detection, and processing a cluster service according to the fault reason. The present application solves the problem in the prior art of cluster service systems being unable to switch a service from a node having an RAID card fault to a normal node for operation as a result of heartbeat networks being unable to detect the RAID card fault in the node.
Need to check novelty before this filing date? Find Prior Art

Description

Cluster service processing method, server and system

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 19, 2023, with application number 202310598620.9 and application name “A Cluster Business Processing Method, Server and System”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of database technology, and in particular to a cluster business processing method, server, and system. Background Art

[0003] The rapid development of digitalization and informatization has brought significant convenience to people's lives and production, but it has also resulted in the generation of massive amounts of data. Existing technologies typically use databases to store and analyze this data. To ensure business continuity, databases are often built using high-availability clusters. When a high-availability cluster detects a failure in one or more nodes within the cluster, it switches services from the failed node to a functioning node, thus avoiding service interruptions.

[0004] In the prior art, node switching in a high-availability cluster usually relies on heartbeat network detection. Heartbeat network detection mainly detects whether a node fails by monitoring the heartbeat signals of the nodes in the cluster. When the heartbeat signal of a node in the cluster is not monitored within a specified time, the node is determined to be failed, and the business running on the node is switched to a normal node. However, if the node is not failed, if the independent redundant disk array (Redundant Array of Independent Disks, referred to as RAID) card used to control data storage in the node fails, the IO flow of multiple disks managed by the RAID card cannot be read and written normally, thereby affecting the database business, and node switching is also required at this time. Since the node has not failed, its heartbeat signal can still be monitored, so the cluster business system will not switch the business from the node where the RAID card fails to the normal node.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a cluster service processing method, server and system for solving the problem in the prior art that the heartbeat network detection cannot detect the RAID card failure in the node, resulting in the cluster service system not switching the service from the node with the RAID card failure to run on the normal node.

[0007] In a first aspect, an embodiment of the present application provides a cluster service processing method, which is applied to a cluster service system, the cluster service system comprising: a first node and a second node; the method comprising: obtaining a detection result of a local input / output (IO) flow detection when the heartbeat network of the first node and the heartbeat network of the second node are normal; obtaining a detection result of a peer IO flow detection when the detection result of the local IO flow detection is abnormal; determining a fault cause based on the detection result of the peer IO flow detection and the detection result of the local IO flow detection, and processing the cluster service according to the fault cause; wherein the local input / output (IO) flow detection is a first local IO flow detection by the first node of an IO flow between a first independent redundant disk array (RAID) card in the first node and a first disk, and a second local IO flow detection by the second node of an IO flow between a second RAID card in the second node and a second disk; the peer IO flow detection is a first peer IO flow detection by the first node of an IO flow between a second RAID card in the second node and a second disk, and / or a second peer IO flow detection by the second node of an IO flow between a first RAID card in the first node and a first disk.

[0008] In the embodiment of the present application, by introducing an IO flow detection service, each node detects the local IO flow respectively. In the case of an abnormal detection result, each node detects the IO flow of the opposite end. Since the local IO flow detection service is abnormal, it may be caused by the local IO flow detection service abnormality or a soft fault in the local RAID card. Therefore, in the case of an abnormal local IO flow detection, by detecting the IO flow of the opposite end, it can be further determined whether the local RAID card has a soft fault. This can solve the problem of the system freezing when the disk cannot read and write normally, and not switching the master and backup nodes of the cluster service system, thereby improving the reliability of the cluster service system.

[0009] In a specific embodiment, obtaining a detection result of a local IO flow detection includes: the first node and the second node respectively initiate a first data read instruction to a disk of the local node; wherein the first data read instruction is used to read first data from the disk in the local node; obtaining first data returned by the disk of the local node based on the first data read instruction; and the detection result of the local IO flow detection being abnormal includes: failing to obtain the first data returned by the disk of the local node.

[0010] In the above embodiment, the first node and the second node respectively send a read data instruction to their respective local disks and return data based on the data read instruction. If the data in the local disk can be obtained, the local IO flow detection result is normal. If the data in the local disk cannot be obtained, the local IO flow detection result is abnormal. Through the above method, each node can realize the detection of the local IO flow.

[0011] In a specific embodiment, obtaining the first data returned by the disk of the local node based on the first read data instruction includes: obtaining the first data returned by the disk of the local node based on the first read data instruction at preset time intervals within a preset first time period; if the detection result of the local IO flow detection is abnormal, it also includes: if the total number of times the first data returned by the disk of the local node is obtained at the preset time intervals within the preset first time period is less than a first number threshold, then determining that the detection result of the local IO flow detection is abnormal.

[0012] In the above embodiment, each node reads the data from the disk of the local node at a predetermined interval within a preset first time period. If the total number of times the first data returned by the disk of the local node is obtained within the preset first time period is greater than or equal to the first number threshold, then the detection result of the local IO flow detection can be determined to be normal; otherwise, it is abnormal. By determining the number of times the first data is successfully obtained within a preset time period, the determination of the detection result of the IO flow detection is made more accurate and reliable.

[0013] In a specific embodiment, obtaining first data returned by a disk of a local node based on a first read data instruction includes: obtaining the first data returned by the disk of the local node based on the first read data instruction within a preset second time period; if a detection result of a local IO flow detection is abnormal, the method further includes: if a difference between an acquisition time of the first data returned by the disk of the local node and a preset second time is greater than a first time threshold, determining that the detection result of the local IO flow detection is abnormal; wherein the preset second time is a maximum time for normally obtaining the first data returned by the disk of the local node.

[0014] In the above embodiment, by determining the timeout time for each node to obtain the first data within the preset second time period, if the timeout time is greater than the first time threshold, the local IO flow detection result is determined to be abnormal, otherwise it is normal. Through the above embodiment, the detection result of the local IO flow detection can be made more reliable.

[0015] In a specific embodiment, obtaining the first data returned by the disk of the local node based on the first read data instruction includes: obtaining the first data returned by the disk of the local node based on the first read data instruction within a preset first time period; the detection result of the local IO flow detection is abnormal, and also includes: within the preset first time period, the number of times that the difference between the acquisition time of the first data returned by the disk of the local node and a preset second time is greater than a first time threshold is greater than a second number threshold; the preset second time is the maximum time for normally obtaining the first data returned by the disk of the local node.

[0016] In the above embodiment, the detection result of the local IO flow is judged by judging the number of timeouts for each node to obtain the first data within the first time period, thereby further improving the reliability and accuracy of the detection.

[0017] In a specific embodiment, obtaining the detection result of the peer IO flow detection includes: the first node or the second node initiating a second read data instruction to the disk of the peer node; wherein the second read data instruction is used to read second data in the disk of the peer node; obtaining second data returned by the disk of the peer node based on the second read data instruction; if the second data returned by the disk of the peer node is obtained, determining that the detection result of the peer IO flow detection is normal; if the second data returned by the disk of the peer node is not obtained, determining that the detection result of the peer IO flow detection is abnormal.

[0018] In the above implementation, each node sends a read data command to the peer node's disk. If the peer node returns the second data, the peer IO flow detection result is considered normal; otherwise, it is considered abnormal. By adding peer IO flow detection, the local IO flow detection results can be further verified, enhancing the accuracy and reliability of fault diagnosis.

[0019] In a specific embodiment, the determining the cause of the fault based on the detection result of the opposite-end IO flow detection and the detection result of the local IO flow detection includes: when the detection result of the first local IO flow detection is abnormal and the detection result of the first opposite-end IO flow detection is normal, determining that the RAID state of the first node is abnormal; when the detection result of the second local IO flow detection is abnormal and the detection result of the second opposite-end IO flow detection is normal, determining that the RAID state of the second node is abnormal; when the detection result of the first local IO flow detection is abnormal, the detection result of the first opposite-end IO flow detection is abnormal, and the detection result of the second local IO flow detection is normal, determining that the IO flow detection service of the first node is abnormal; when the detection result of the second local IO flow detection is abnormal, the detection result of the second opposite-end IO flow detection is abnormal, and the detection result of the first local IO flow detection is normal, determining that the IO flow detection service of the second node is abnormal.

[0020] In the above embodiment, when the local IO flow detection result is abnormal, the fault cause can be accurately determined by obtaining the detection result of the peer IO flow and combining it with the local IO flow detection result, thereby achieving accurate judgment of the cluster service system fault.

[0021] In a specific embodiment, the method also includes: when the heartbeat detection result of the first node is abnormal and the heartbeat detection result of the second node is normal, determining that the heartbeat network of the first node is abnormal; when the heartbeat detection result of the first node is normal and the heartbeat detection result of the second node is abnormal, determining that the heartbeat network of the second node is abnormal; when the heartbeat detection results of the first node and the second node are both abnormal, determining that the heartbeat networks of the first node and the second node are both abnormal.

[0022] In the above implementation manner, in addition to the system pseudo-death situation of each node, the cluster business system may also have other faults. By detecting the heartbeat network of each node and judging whether the heartbeat network of each node is normal based on the detection results, the fault judgment of the cluster business system can be more accurate.

[0023] In a specific embodiment, the processing of the cluster service according to the cause of the fault includes: the cluster service system is in a hot standby scenario, the first node is the master node, and the second node is the standby node; when it is determined that the RAID card status of the first node is abnormal and the RAID card status of the second node is normal, the cluster service is switched from the first node to the second node for operation, and an alarm processing of a soft fault of the RAID card of the first node is performed; when it is determined that the RAID card status of the second node is abnormal and the RAID card status of the first node is normal, an alarm processing of a soft fault of the RAID card of the second node is performed; when it is determined that the RAID card status of the first node is abnormal and the RAID card status of the second node is abnormal, an alarm processing of a soft fault of the RAID cards of the first node and the second node is performed; when it is determined that the IO flow detection service of the first node and / or the second node is abnormal, an alarm processing of an IO flow detection failure of the first node and / or the second node is performed.

[0024] In the above implementation, when the system is in a hot standby scenario, the cluster service system is processed accordingly based on each failure cause, thereby improving the reliability of the system operation.

[0025] In a specific embodiment, the processing of the cluster service according to the cause of the fault includes: the cluster service system is in a dual-active scenario, the first node and the second node are backup nodes for each other, the first cluster service runs on the first node, and the second cluster service runs on the second node; when it is determined that the RAID card status of the first node is abnormal and the RAID card status of the second node is normal, the first cluster service is switched from the first node to the second node for operation, and an alarm processing of a soft fault of the RAID card of the first node is performed; when it is determined that the RAID card status of the second node is abnormal and the RAID card status of the first node is normal, the second cluster service is switched from the second node to the first node for operation, and an alarm processing of a soft fault of the RAID card of the second node is performed; when it is determined that the RAID card status of the first node is abnormal and the RAID card status of the second node is abnormal, an alarm processing of a soft fault of the RAID card of the first node and the second node is performed; when it is determined that the IO flow detection service of the first node and / or the second node is abnormal, an alarm processing of an IO flow detection fault of the first node and / or the second node is performed.

[0026] In the above implementation, when the system is in an active-active scenario, the cluster service system is processed accordingly based on each fault cause, thereby improving the reliability of the system operation.

[0027] In a second aspect, an embodiment of the present application provides a server comprising: a processor, a memory, and a communication interface; the memory is used to store executable instructions of the processor; wherein the processor is configured to execute the cluster business processing method described in the first aspect by executing the executable instructions.

[0028] In a third aspect, an embodiment of the present application provides a cluster service system, comprising: at least one first node and at least one second node, the first node being a master node and the second node being a backup node; wherein the first node executes the cluster service processing method described in the first aspect.

[0029] Embodiments of the present application provide a cluster service processing method, server, and system. The method is applied to a cluster service system, the cluster service system including a first node and a second node. The method includes: obtaining a detection result of a local input / output (IO) flow detection when the heartbeat network of the first node and the heartbeat network of the second node are normal; obtaining a detection result of a peer IO flow detection when the local IO flow detection result is abnormal; determining a fault cause based on the detection result of the peer IO flow detection and the detection result of the local IO flow detection, and processing the cluster service according to the fault cause; wherein the local input / output (IO) flow detection is a first local IO flow detection performed by the first node on an IO flow between a first independent redundant array of disks (RAID) card in the first node and a first disk, and a second local IO flow detection performed by the second node on an IO flow between a second RAID card in the second node and a second disk; and the peer IO flow detection is a first peer IO flow detection performed by the first node on an IO flow between a second RAID card in the second node and a second disk, and / or a second peer IO flow detection performed by the second node on an IO flow between a first RAID card in the first node and the first disk. Compared with the existing technology that relies on heartbeat network detection to realize the switching of services from a faulty node to a normal node, the present application determines the cause of the failure of the service cluster based on the local IO flow detection and the peer IO flow detection of the first node and the second node, combined with the heartbeat detection results of the first node and the second node, and processes the cluster service according to the cause of the failure, so that the service cluster of the present application can switch the service from the faulty node to the normal node in time when the service needs to be switched due to a soft failure of the RAID card, solving the problem in the existing technology that the heartbeat network detection cannot detect the soft failure of the RAID card in the node, resulting in the cluster service system not switching the service from the node with the RAID card failure to run normally. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0031] Figure 1 is a schematic diagram of the structure of the cluster service system;

[0032] FIG2 is a flow chart of a first embodiment of a cluster service processing method according to an embodiment of the present application;

[0033] FIG3 is a schematic diagram of IO flow detection performed by a cluster service system;

[0034] FIG4 is a flow chart of a second embodiment of a cluster service processing method provided in an embodiment of the present application;

[0035] FIG5 is a flow chart of a third embodiment of a cluster service processing method provided in an embodiment of the present application;

[0036] FIG6 is a flow chart of a fourth embodiment of a cluster service processing method provided in an embodiment of the present application;

[0037] FIG7 is a flow chart of a fifth embodiment of a cluster service processing method provided in an embodiment of the present application;

[0038] FIG8 is a schematic structural diagram of a server embodiment provided in an embodiment of the present application;

[0039] FIG9 is a schematic structural diagram of another server embodiment provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] To make the purpose, technical solutions, and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments made by ordinary technicians in this field based on the inspiration of these embodiments fall within the scope of protection of this application.

[0041] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the numbers used in this way are interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0042] First, let’s explain the terms involved in this application:

[0043] RAID card: A Redundant Array of Independent Disks (RAID) combines multiple independent disks into a single, high-capacity disk group. A RAID card manages the multiple disks that make up the disk array. When implementing database services, the operating system uses the RAID card to read and write data to the disks it manages.

[0044] Figure 1 is a schematic diagram of the structure of a cluster service system. The cluster service system includes a first node 11 and a second node 12. The first node 11 is a master node, and the second node 12 is a backup node. A heartbeat network detection is performed between the first node 11 and the second node 12 via a heartbeat network link 13. The heartbeat network detection mainly detects whether a node fails by monitoring the heartbeat signal of the node in the cluster. Specifically, the first node 11 and the second node 12 send heartbeat messages to each other at a fixed frequency via the heartbeat network link 13, and receive heartbeat messages from the opposite node. If the second node 12 does not receive the heartbeat message from the first node 11 within a specified time, it is determined that the first node 11 has failed, and the service running on the first node 11 is switched to the second node 12.

[0045] As shown in Figure 1, the first node 11 and the second node 12 each include a processor running an operating system, a RAID card, and multiple disks managed by the RAID card. To implement database services, the operating system must read and write data to the disks it manages through the RAID card. If the RAID card used to control data reading and writing in a node fails, for example, due to a RAID card program error, the I / O flows of the multiple disks managed by the RAID card will not be able to read and write normally, thus affecting database services. In this case, a node switch is also required. However, since the node has not failed and its heartbeat signal can still be monitored, the cluster service system will not switch services from the node with the RAID card failure to a normal node.

[0046] Based on the above technical problems, the technical conception process of the present application is as follows: how to detect the RAID card failure of a node to switch the cluster service from the node with the RAID card failure to a normal node.

[0047] The technical solution of the present application is described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0048] FIG2 is a flow chart of a first embodiment of a cluster service processing method provided by an embodiment of the present application. The method is applied to a cluster service system, which includes a first node and a second node. Referring to FIG2 , the cluster service processing method specifically includes the following steps:

[0049] Step S201: When the heartbeat network of the first node and the heartbeat network of the second node are normal, a detection result of a local input and output IO flow detection is obtained.

[0050] In one example, a cluster service system in a hot standby scenario includes a first node and a second node, where the first node is the primary node and the second node is the backup node. Database services run on the first node. When the first node fails, the database services switch from the first node to the second node. The cluster service system may also include a management node. Storage devices are attached to the first and second nodes. Figure 3 is a schematic diagram of I / O flow detection in the cluster service system.

[0051] The local IO flow detection is a first local IO flow detection performed by a first node on the IO flow between the first independent redundant disk array RAID card and the first disk in the first node, and / or a second local IO flow detection performed by a second node on the IO flow between the second RAID card and the second disk in the second node. As shown in Figure 3, the first local IO flow detection is a first node detection of the IO flow between the first RAID card and the first disk in the first node; the second local IO flow detection is a second node detection of the IO flow between the second RAID card and the second disk in the second node. Local IO flow detection is primarily used to detect whether the local system disk has issues with normal reading and writing.

[0052] In a database service cluster, the first node is the master node and the second node is the backup node. Both the first and second nodes include a RAID card and multiple disks managed by it. The disks can be system disks. To implement database services, the operating system must read and write data to the disks it manages through the RAID card. During the data reading and writing process, an IO flow is formed between the RAID card and the disks. Therefore, by detecting the IO flow between the RAID card and the disks, it can be determined whether the RAID card has experienced a soft fault such as a program execution error. When the detection result of the IO flow between the RAID card and the disks is abnormal, it can be determined that the RAID card has experienced a soft fault such as a program execution error.

[0053] In this embodiment, when the heartbeat network of the first node and the heartbeat network of the second node are normal, the detection results of the local input and output IO flow detection are obtained. Specifically, the first node detects the IO flow between the first RAID card and the first disk in the first node to obtain the detection result of the first local IO flow detection; the second node detects the IO flow between the second RAID card and the second disk in the second node to obtain the detection result of the second local IO flow detection. The first node, acting as the master node, obtains the detection results of the first local IO flow detection and the second local IO flow detection. The detection results of the first local IO flow detection and the second local IO flow detection can also be obtained by the management node.

[0054] Step S202: When the detection result of the local IO flow detection is abnormal, the detection result of the peer IO flow detection is obtained.

[0055] The peer IO flow detection is a first peer IO flow detection of the IO flow between the second RAID card in the second node and the second disk, and / or a second peer IO flow detection of the IO flow between the first RAID card in the first node and the first disk. As shown in Figure 3, the first peer IO flow detection is a first node detection of the IO flow between the second RAID card in the second node and the second disk; the second peer IO flow detection is a second node detection of the IO flow between the first RAID card in the first node and the first disk. The peer IO flow detection is used to verify the results to ensure consistency and reliability, and to verify whether there are any abnormalities in the IO flow detection service itself.

[0056] In this embodiment, if the result of the local IO flow detection is abnormal, the result of the peer IO flow detection is obtained. Specifically, the first node detects the IO flow between the second RAID card and the second disk in the second node to obtain the result of the first peer IO flow detection; the second node also detects the IO flow between the first RAID card and the first disk in the first node to obtain the result of the second peer IO flow detection. The first node, acting as the master node, obtains the results of the first peer IO flow detection and the second peer IO flow detection. Alternatively, the management node may obtain the results of the first peer IO flow detection and the second peer IO flow detection.

[0057] In this embodiment, the first node and the second node also perform heartbeat detection, sending heartbeat messages to each other at a fixed frequency, and receiving heartbeat messages from the peer node to obtain heartbeat detection results. The first node, acting as the master node, obtains the heartbeat detection results of the first node and the second node. Alternatively, the management node can obtain the heartbeat detection results of the first node and the second node. When the heartbeat network of the first node and the heartbeat network of the second node are normal, steps S201 to S202 are performed.

[0058] Step S203: Determine the cause of the fault based on the detection result of the peer IO flow detection and the detection result of the local IO flow detection, and process the cluster service according to the cause of the fault.

[0059] In this embodiment, the first node or the management node determines the cause of the fault based on the detection result of the peer IO flow detection and the detection result of the local IO flow detection.

[0060] For example, under the premise that the heartbeat detection results of the first node and the second node are normal, the detection result of the first local IO flow detection is abnormal, and the detection result of the first peer IO flow detection is normal, it can be preliminarily determined that the RAID card status of the first node is abnormal, and the detection result of the second local IO flow detection is normal, which means that the IO detection service of the second node is normal. When the detection result of the second peer IO flow detection is abnormal, it can be further determined that the RAID status of the first node is abnormal, and the RAID status of the second node is determined to be normal.

[0061] The first node is the master node, the second node is the standby node, and the RAID status of the first node is abnormal while the RAID status of the second node is normal. In this case, the cluster service running on the first node needs to be switched to the second node.

[0062] In one example, a cluster service system operates in an active-active scenario and includes a first node and a second node, with the first and second nodes acting as backup nodes for each other. Database service A runs on the first node, and database service B runs on the second node. If the first node fails, database service A switches from the first node to the second node; if the second node fails, database service B switches from the second node to the first node. The cluster service system may also include a management node.

[0063] If the heartbeat network of the first node and the heartbeat network of the second node are normal, the first node, the second node, or the management node can obtain the results of the local input / output (IO) flow detection. If the local IO flow detection result is abnormal, the results of the peer IO flow detection are obtained. Based on the results of the peer IO flow detection and the local IO flow detection, the first node, the second node, or the management node determines the cause of the failure and processes the cluster service based on the cause of the failure.

[0064] In one example, when the database service is started, the detection results of the first local IO flow detection and the first peer IO flow detection, the detection results of the second local IO flow detection and the second peer IO flow detection, and the heartbeat detection results of the first node and the second node are obtained. According to the detection results of the first local IO flow detection, the first peer IO flow detection, the second local IO flow detection and the second peer IO flow detection, and the heartbeat detection results of the first node and the second node, the RAID card status of the first node and the second node are determined respectively to determine whether to switch the cluster service.

[0065] In one example, after the database service is started, the detection results of the first local IO flow detection and the first peer IO flow detection, the detection results of the second local IO flow detection and the second peer IO flow detection, and the heartbeat detection results of the first node and the second node can be re-obtained at preset intervals. Based on the detection results of the first local IO flow detection, the first peer IO flow detection, the second local IO flow detection and the second peer IO flow detection, and the heartbeat detection results of the first node and the second node, the RAID card status of the first node and the second node can be determined respectively to determine whether to switch the cluster service.

[0066] In one example, the detection results of the first local IO flow detection and the first peer IO flow detection, the detection results of the second local IO flow detection and the second peer IO flow detection, and the heartbeat detection results of the first node and the second node may be obtained multiple times within a preset time. After determining the RAID card status of the first node and the second node based on the multiple detection results of the first local IO flow detection, the first peer IO flow detection, the second local IO flow detection, and the second peer IO flow detection, as well as the multiple heartbeat detection results of the first node and the second node, the number of abnormal RAID card statuses of the first node and the second node is counted, and based on the number of abnormal RAID card statuses of the first node and the second node, it is determined whether to switch the cluster service. Exemplarily, the first node is the master node and the second node is the backup node. If the number of abnormal RAID card statuses of the first node exceeds a threshold, and the number of abnormal RAID card statuses of the second node is 0, that is, if the multiple detection results of the RAID card of the second node all indicate a normal status, then the cluster service is switched.

[0067] In this embodiment, a cluster service system includes a first node and a second node; when the heartbeat network of the first node and the heartbeat network of the second node are normal, a detection result of a local input / output (IO) flow detection is obtained; when the detection result of the local IO flow detection is abnormal, a detection result of a peer IO flow detection is obtained; based on the detection result of the peer IO flow detection and the detection result of the local IO flow detection, a fault cause is determined, and the cluster service is processed according to the fault cause; wherein the local IO flow detection is a first local IO flow detection of the IO flow between the first independent redundant disk array (RAID) card in the first node and the first disk, and / or a second local IO flow detection of the IO flow between the second RAID card in the second node and the second disk; the peer IO flow detection is a first peer IO flow detection of the IO flow between the second RAID card in the second node and the second disk, and / or a second peer IO flow detection of the IO flow between the first RAID card in the first node and the first disk, performed by the second node. Compared with the existing technology that relies on heartbeat network detection to realize the switching of services from a faulty node to a normal node, the present application determines the cause of the failure of the service cluster based on the local IO flow detection and the peer IO flow detection of the first node and the second node, combined with the heartbeat detection results of the first node and the second node, and processes the cluster service according to the cause of the failure, so that the service cluster of the present application can switch the service from the faulty node to the normal node in time when the service needs to be switched due to a soft failure of the RAID card, solving the problem in the existing technology that the heartbeat network detection cannot detect the soft failure of the RAID card in the node, resulting in the cluster service system not switching the service from the node with the RAID card failure to run normally.

[0068] Figure 4 is a flow chart of a second embodiment of a cluster business processing method provided by an embodiment of the present application. Based on the embodiment shown in Figure 2 above, the above step S201 may include: the first node and the second node respectively initiate a first read data instruction to the disk of the local node; wherein the first read data instruction is used to read the first data in the disk in the local node; obtain the first data returned by the disk of the local node based on the first read data instruction; the detection result of the local IO flow detection is abnormal, including: the first data returned by the disk of the local node is not obtained.

[0069] Specifically, step S201 may include the following steps:

[0070] Step S401: a first node initiates a first data read instruction to a first disk in the first node; the first data read instruction is used to read first data in the first disk.

[0071] Step S402: The first node obtains the first data returned by the first disk based on the first data read instruction; if the first data returned by the first disk is obtained, it is determined that the detection result of the first local IO stream detection is normal; if the first data returned by the first disk is not obtained, it is determined that the detection result of the first local IO stream detection is abnormal.

[0072] In this embodiment, the RAID card manages multiple disks. When implementing database services, the operating system needs to read and write data on the disks it manages through the RAID card. During the process of reading and writing data, an IO flow is formed between the RAID card and the disk. Therefore, it is possible to determine whether a soft fault has occurred in the RAID card by detecting the IO flow between the RAID card and the disk. When a soft fault such as a program error occurs in the RAID card, the disk will not respond to the read instruction and return data. Therefore, the IO flow between the RAID card and the system disk can be detected by issuing a read data instruction to the disk, thereby determining whether a soft fault has occurred in the RAID card. For example, the underlying IO testing tools, such as the disk stress tester (Flexible Input Output tester, referred to as FIO), IO testing software (Input Output meter, referred to as IOmeter), etc., can be called to perform a read operation on the disk.

[0073] Specifically, when the first node performs a first local IO flow detection, it initiates a first read data instruction to the first disk in the first node. The first read data instruction is used to read the first data in the first disk managed by the first RAID card. If the RAID card in the first node is in normal status, the first disk will return the first data to the operating system of the first node based on the first read data instruction; if a soft failure occurs in the RAID card in the first node, the first disk will not return data.

[0074] The first node obtains the first data returned by the first disk based on the first read data instruction. If the first data returned by the first disk is successfully obtained, it is determined that the detection result of the first local IO stream detection is normal; if the first data returned by the first disk is not obtained, it is determined that the detection result of the first local IO stream detection is abnormal.

[0075] Based on the embodiment shown in Figure 2 above, the above step S202 may include: the first node or the second node initiates a second read data instruction to the disk in the opposite node; wherein the second read data instruction is used to read second data in the disk in the opposite node; obtaining second data returned by the disk in the opposite node based on the second read data instruction; if the second data returned by the disk in the opposite node is obtained, determining that the detection result of the opposite IO flow detection is normal; if the second data returned by the disk in the opposite node is not obtained, determining that the detection result of the opposite IO flow detection is abnormal.

[0076] Specifically, step S202 may include the following steps:

[0077] Step S403: the first node initiates a second data read instruction to the second disk in the second node; the second data read instruction is used to read second data in the second disk.

[0078] Step S404: The first node obtains the second data returned by the second disk based on the second data read instruction; if the second data returned by the second disk is obtained, it is determined that the detection result of the first peer IO flow detection is normal; if the second data returned by the second disk is not obtained, it is determined that the detection result of the first peer IO flow detection is abnormal.

[0079] It should be noted that there is no specific limitation on the execution order of steps S402 to S404.

[0080] In this embodiment, the first node also performs I / O flow detection on the peer node, namely, first peer I / O flow detection. Specifically, a second read data instruction is issued to the second disk in the second node. This second read data instruction is used to read the second data from the second disk. If the RAID card in the second node is functioning properly, the second disk returns the second data to the operating system of the first node based on the second read data instruction. If the RAID card in the second node experiences a soft failure, the second disk does not return the data.

[0081] The first node obtains the second data returned by the second disk based on the second read data instruction; if the second data returned by the second disk is successfully obtained, it is determined that the detection result of the first peer IO flow detection is normal; if the second data returned by the second disk is not obtained, it is determined that the detection result of the first peer IO flow detection is abnormal.

[0082] In this embodiment, the attribute parameters of the IO flow detection include: interval time, timeout time, timeout times, etc. Exemplarily, the first node may perform local IO flow detection and peer IO flow detection at every preset interval time. For example, within a preset first time period, the first node obtains the first data returned by the first disk according to a preset time interval. If the first data returned by the first disk can be obtained at each time interval point within the preset first time period, the detection result of the first local IO flow detection is determined to be normal; or, if the number of times the first node obtains the first data returned by the first disk at all time intervals within the preset first time period is greater than or equal to a first number threshold, the detection result of the first local IO flow detection is determined to be normal; if the first node cannot obtain the first data returned by the first disk at all time intervals within the preset first time period, the detection result of the first local IO flow detection is determined to be abnormal; or if the number of times the first node obtains the first data returned by the first disk at all time intervals within the preset first time period is less than a first number threshold, the detection result of the first local IO flow detection is determined to be abnormal; wherein the first number threshold is the minimum number of times the first node can obtain the first data returned by the first disk within the preset time period; the interval time can be 1s, 5s, 1min, 1h, etc., and is not particularly limited here.

[0083] In some implementations, if the difference between the time when the first node obtains the first data returned by the first disk and a preset first time is less than a first time threshold, the detection result of the first local IO flow detection is determined to be normal; if the difference between the time when the first node obtains the first data returned by the first disk and a preset second time is greater than the first time threshold, the detection result of the first local IO flow detection is determined to be abnormal, where the second time is a preset maximum time within which the first node can normally obtain the first data returned by the first disk;

[0084] In some implementations, if the number of timeouts for the first node to obtain the first data returned by the first disk within a preset first time period is greater than a second threshold, the result of the first local IO flow detection is determined to be abnormal; and if the number of timeouts for the first node to obtain the first data returned by the first disk is less than or equal to the second threshold, the result of the first local IO flow detection is determined to be normal. The number of timeouts is the maximum number of times, within the time period, that the difference between the time the first node obtained the first data returned by the first disk and the preset second time was greater than the first time threshold.

[0085] Similarly, the second node performs the second local IO flow detection and the second peer IO flow detection, which can be performed with reference to the above steps S401 to S404.

[0086] In this embodiment, the first node performs a first local IO flow detection and a first peer IO flow detection by issuing a read data command to the first and second disks in the first node, respectively, to read data from the first and second disks. This allows the operation of the local and peer RAID cards to be checked, respectively. This allows the IO flows of the local and peer nodes to be checked to determine whether a soft failure has occurred in the RAID, without affecting database services. This ensures that the service cluster can promptly switch services from the faulty node to a healthy node when a soft failure in the RAID card causes a service switchover.

[0087] FIG5 is a flow chart of a third embodiment of a cluster service processing method provided by an embodiment of the present application. Based on the embodiments shown in FIG2 to FIG4 above, step S203 specifically includes the following steps:

[0088] Step S501: Determine the RAID card status of the first node and the second node respectively according to the detection results of the first local IO flow detection, the first peer IO flow detection, the second local IO flow detection, the second peer IO flow detection, and the heartbeat detection results of the first node and the second node.

[0089] Among them, one of the first node and the second node is a master node, and the other node is a backup node.

[0090] In this embodiment, the RAID card status of the first node and the second node is determined based on the detection results of the first local IO flow detection, the first peer IO flow detection, the second local IO flow detection, the second peer IO flow detection, and the heartbeat detection results of the first node and the second node.

[0091] Local IO flow detection is mainly used to detect whether the local system disk has problems with normal reading and writing. The peer IO flow detection is used to verify the local IO flow detection results of the peer node to ensure the consistency and reliability of the results, and to verify whether there are any abnormalities in the IO flow detection service itself.

[0092] Specifically, when the heartbeat detection results of the first node and the second node are both normal, the detection result of the first local IO flow detection is abnormal, and the detection result of the first peer IO flow detection is normal, it is determined that the RAID status of the first node is abnormal. If the heartbeat detection results of the first node and the second node are normal, it means that the network of the first node and the second node is normal; if the detection result of the first local IO flow detection is abnormal, it means that the local disk of the first node may have a problem of not being able to read and write normally (it may also be that the IO flow detection service of the first node is abnormal). In order to further verify whether the local disk of the first node has a problem of not being able to read and write normally, it is necessary to judge the first peer IO flow detection result. If the first peer IO flow detection result is normal, the situation of the first node IO flow detection service being abnormal can be ruled out. Therefore, it can be determined that the RAID status of the first node is abnormal.

[0093] When the heartbeat detection results of the first node and the second node are both normal and the detection result of the first local IO flow detection is normal, it can be preliminarily determined that the RAID card of the first node and the IO detection service of the first node are normal. In order to further verify the above conclusion, it can be judged by the result of the second peer end IO flow detection. If the detection result of the second peer end IO flow detection is normal, it is determined that the RAID status of the first node is normal.

[0094] Similarly, the heartbeat detection results of the first node and the second node are both normal, and the first local IO flow detection is normal, and the second peer IO flow detection is normal. If the detection result of the first peer IO flow detection is abnormal, it can be preliminarily determined that the RAID card of the second node is abnormal. At this time, if the detection result of the second local IO flow detection is abnormal, it is further determined that the RAID status of the second node is abnormal. If the heartbeat detection results of the first node and the second node are normal, it means that the network of the first node and the second node is normal; if the first local IO flow detection is normal, and the second peer IO flow detection is normal, it can be respectively explained that the IO flow detection services of the first node and the second node are normal. At this time, if the detection result of the second local IO flow detection is abnormal, it can be preliminarily determined that the local disk of the second node has a problem of not being able to read and write normally. In order to further verify this conclusion, the detection result of the first peer IO flow detection is used for judgment. If the detection result is abnormal, the RAID card status of the second node is abnormal, which further verifies that the local disk of the second node has a problem of not being able to read and write normally. Therefore, it can be further determined that the RAID status of the second node is abnormal.

[0095] If the heartbeat detection results of the first node and the second node are both normal and the detection result of the second local IO flow detection is normal, it can be preliminarily determined that the local disk reading and writing of the second node is normal. In order to further verify the above conclusion, the detection result of the first peer IO flow detection is verified. If the above result is normal, it is determined that the RAID status of the second node is normal.

[0096] Specifically, if the heartbeat detection results of the first node and the second node are both normal, the detection result of the first local IO flow detection is abnormal, the detection result of the first peer IO flow detection is abnormal, the detection result of the second local IO flow detection is normal, and the detection result of the second peer IO flow detection is normal, then it is determined that the IO flow detection service of the first node is abnormal. If the heartbeat detection results of the first node and the second node are normal, it means that the network between the first node and the second node is normal; if the detection result of the first local IO flow detection is abnormal, it indicates that the local disk of the first node may not be able to read and write normally (or it may be that the IO flow detection service of the first node is abnormal), but the detection result of the second peer IO flow detection is normal, it means that the local system disk of the first node can be read and written normally. In this case, the abnormal status of the RAID card of the first node can be ruled out, and it can be determined that the abnormal detection result of the first local IO flow detection is due to an abnormality in the IO flow detection service of the first node; the detection result of the second local IO flow detection is normal, indicating that the local disk of the second node can be read and written normally, but the detection result of the first peer IO flow detection is abnormal, further verifying that the IO flow detection of the first node is abnormal.

[0097] Similarly, when the heartbeat detection results of the first node and the second node are both normal, the detection result of the first local IO flow detection is normal, the detection result of the first peer IO flow detection is normal, the detection result of the second local IO flow detection is abnormal, and the detection result of the second peer IO flow detection is abnormal, it is determined that the IO flow detection service of the second node is abnormal.

[0098] In some implementations, the network conditions of the first node and the second node may also affect the detection result of the peer IO flow detection. When the heartbeat detection result of the first node and / or the second node is abnormal, the detection result of the peer IO flow detection is also abnormal.

[0099] Step S502: When it is determined that the RAID card status of the master node is abnormal and the RAID card status of the backup node is normal, the cluster service is switched from the master node to the backup node for operation, and an alarm processing of the soft fault of the master node RAID card is performed.

[0100] Step S503: When the cluster service system is in a hot standby scenario and it is determined that the RAID card status of the standby node is abnormal and the RAID card status of the master node is normal, an alarm processing of a soft fault of the RAID card of the standby node is performed;

[0101] In a cluster service system operating in an active-active scenario, if the RAID card status of the standby node is determined to be abnormal and the RAID card status of the active node is normal, the cluster service is switched from the standby node to the active node, and an alarm for a soft fault on the standby node's RAID card is processed.

[0102] Step S504: When it is determined that the RAID card status of the master node is abnormal and the RAID card status of the backup node is abnormal, an alarm process of a soft fault of the RAID cards of the master node and the backup node is performed.

[0103] Step S505: When it is determined that the IO flow detection of the primary node and / or the backup node is abnormal, an alarm process of the IO flow detection failure of the primary node and / or the backup node is performed.

[0104] Specifically, an IO flow detection anomaly indicates that an IO test tool, such as FIO or IOmeter, has failed. An alarm needs to be issued for the IO flow detection failure so that the management personnel can handle the IO flow detection failure.

[0105] In one example, the cluster business system is in a hot standby scenario, including a first node and a second node, the first node is a master node, the second node is a backup node, and the database business runs on the first node. When the first node fails, the database business switches from the first node to the second node. The cluster business system may also include a management node. The first node or the management node determines the RAID card status of the first node and the second node respectively based on the detection results of the first local IO flow detection, the first peer IO flow detection, the second local IO flow detection and the second peer IO flow detection, and the heartbeat detection results of the first node and the second node, and processes the cluster business according to the RAID card status of the first node and the second node. The first node is the master node and the second node is the backup node. The specific method is shown in Table 1 below:

[0106] Table 1 IO flow detection results and processing solutions for the first and second nodes in the hot standby scenario

[0107] In one example, a cluster service system operates in an active-active scenario and includes a first node and a second node, with the first and second nodes acting as backup nodes for each other. A first cluster service runs on the first node, and a second cluster service runs on the second node. If the first node fails, the first cluster service switches from the first node to the second node; if the second node fails, the second cluster service switches from the second node to the first node. The cluster service system may also include a management node.

[0108] The first node, the second node, or the management node obtains the detection results of the first local IO flow detection and the first peer IO flow detection, the detection results of the second local IO flow detection and the second peer IO flow detection, and the heartbeat detection results of the first node and the second node. The first node, the second node, or the management node determines the RAID card status of the first node and the second node respectively based on the detection results of the first local IO flow detection, the first peer IO flow detection, the second local IO flow detection, and the second peer IO flow detection, as well as the heartbeat detection results of the first node and the second node, and processes the cluster service based on the RAID card status of the first node and the second node. The specific method is shown in Table 2 below:

[0109] Table 2 IO flow detection results and processing solutions for the first and second nodes in the active-active scenario

[0110] In the active-active scenario, the RAID status of the first node is abnormal, while the RAID status of the second node is normal. FIG6 is a flow chart of a fourth embodiment of a cluster service processing method provided by an embodiment of the present application. As shown in FIG6 , the first node and the second node are each other's backup nodes, database service A runs on the first node, and database service B runs on the second node. The heartbeat detection results of the first node and the second node are both normal, the detection result of the first local IO flow detection is abnormal, the detection result of the first peer IO flow detection is normal, the detection result of the second local IO flow detection is normal, and the detection result of the second peer IO flow detection is abnormal. It is determined that the system disk reading of the first node is abnormal, and there is a system pseudo-death situation. The database service A needs to be switched from the first node to the second node.

[0111] In addition, since a soft failure of the RAID card may cause problems in the operation of the operating system, which in turn may cause the cluster service to be unable to switch normally, it is necessary to determine whether the cluster service has been successfully switched. If the switch is successful, the switching process ends; if the switch fails, the first node can restart the second node through the out-of-band management channel to switch the cluster service from the second node to the first node. Exemplarily, the out-of-band management channel can be an Intelligent Platform Management Interface (IPMI) channel. Figure 7 is a flow chart of a fifth embodiment of a cluster service processing method provided in an embodiment of the present application.

[0112] As shown in Figure 7, the embodiment of the present application performs IO flow detection by calling the underlying IO test tools, such as FIO, IOmeter, etc., to perform regular read operations on the system disk. For example, if the result is returned, it is normal; if there is no response after the timeout, it is determined that the system disk reading is abnormal and there is a system pseudo-death situation. The result needs to be synchronized to the node where the cluster business is located to enable the cluster business to switch to the normal node to run and ensure business continuity. If the node where the cluster business is located at this time cannot be switched under the pseudo-death system, a network channel, such as an IPMI channel, can be established through out-of-band management of the normal node and the faulty node to restart the faulty node and thus achieve the switching of the cluster business.

[0113] In this embodiment, the first node determines the RAID card status of the first and second nodes based on the detection results of the first local IO flow detection, the first peer IO flow detection, the second local IO flow detection, and the second peer IO flow detection, as well as the heartbeat detection results of the first and second nodes. Cluster services are then processed based on the RAID card status of the first and second nodes. When a service switch is required due to a RAID card soft failure, the service can be promptly switched from the faulty node to a normal node, thereby providing an alarm for abnormal conditions. This further solves the problem in the prior art where services cannot be switched from a node with a faulty RAID card to a normal node due to the inability of heartbeat network detection to detect a RAID card soft failure in the node.

[0114] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0115] FIG8 is a schematic structural diagram of a server embodiment provided in an embodiment of the present application; as shown in FIG8 , the server 60 includes: an acquisition module 61 and a processing module 62 . The acquisition module 61 is used to obtain the detection result of the local input and output IO flow detection when the heartbeat network of the first node and the heartbeat network of the second node are normal; the acquisition module 61 is also used to obtain the detection result of the peer IO flow detection when the detection result of the local IO flow detection is abnormal; the processing module 62 is used to determine the cause of the fault based on the detection result of the peer IO flow detection and the detection result of the local IO flow detection, and process the cluster service according to the cause of the fault; wherein the local IO flow detection is a first local IO flow detection of the IO flow between the first independent redundant disk array RAID card in the first node and the first disk, and / or a second local IO flow detection of the IO flow between the second RAID card in the second node and the second disk; the peer IO flow detection is a first peer IO flow detection of the IO flow between the second RAID card in the second node and the second disk, and / or a second peer IO flow detection of the IO flow between the first RAID card in the first node and the first disk.

[0116] The server provided in the embodiment of the present application can execute the technical solution shown in the above method embodiment. Its implementation principles and beneficial effects are similar and will not be repeated here.

[0117] In a possible implementation scheme, the acquisition module 61 is specifically configured to cause the first node and the second node to respectively initiate a first data read instruction to the disk of the local node; wherein the first data read instruction is used to read first data from the disk of the local node; obtain first data returned by the disk of the local node based on the first data read instruction; and the detection result of the local IO flow detection is abnormal, including: the first data returned by the disk of the local node is not obtained.

[0118] In a possible implementation scheme, the acquisition module 61 is specifically used to obtain the first data returned by the disk of the local node based on the first read data instruction at a preset time interval within a preset first time period; if the detection result of the local IO flow detection is abnormal, it also includes: if the total number of times the first data returned by the disk of the local node is obtained at the preset time interval within the preset first time period is less than a first number threshold, then the detection result of the local IO flow detection is determined to be abnormal.

[0119] In a possible implementation scheme, the acquisition module 61 is specifically used to obtain, within a preset second time period, the first data returned by the disk of the local node based on the first read data instruction; if the detection result of the local IO flow detection is abnormal, it also includes: if the difference between the acquisition time of the first data returned by the disk of the local node and the preset second time is greater than a first time threshold, then determining that the detection result of the local IO flow detection is abnormal; wherein the preset second time is the maximum time for normally obtaining the first data returned by the disk of the local node.

[0120] In a possible implementation scheme, the acquisition module 61 is specifically used to obtain the first data returned by the disk of the local node based on the first read data instruction within a preset first time period; the detection result of the local IO flow detection is abnormal, and also includes: within the preset first time period, the number of times the difference between the acquisition time of the first data returned by the disk of the local node and a preset second time is greater than a first time threshold is greater than a second number threshold; the preset second time is the maximum time for normally obtaining the first data returned by the disk of the local node.

[0121] The server provided in the embodiment of the present application can execute the technical solution shown in the above method embodiment. Its implementation principles and beneficial effects are similar and will not be repeated here.

[0122] In a possible implementation scheme, the acquisition module 61 is specifically used for the first node or the second node to initiate a second read data instruction to the disk in the opposite node; wherein the second read data instruction is used to read the second data in the disk in the opposite node; obtain the second data returned by the disk in the opposite node based on the second read data instruction; if the second data returned by the disk in the opposite node is obtained, it is determined that the detection result of the opposite IO flow detection is normal; if the second data returned by the disk in the opposite node is not obtained, it is determined that the detection result of the opposite IO flow detection is abnormal.

[0123] The server provided in the embodiment of the present application can execute the technical solution shown in the above method embodiment. Its implementation principles and beneficial effects are similar and will not be repeated here.

[0124] In a possible implementation scheme, the processing module 62 is specifically configured to determine that the RAID status of the first node is abnormal when the detection result of the first local IO flow detection is abnormal and the detection result of the first peer-end IO flow detection is normal; determine that the RAID status of the second node is abnormal when the detection result of the second local IO flow detection is abnormal and the detection result of the second peer-end IO flow detection is normal; determine that the IO flow detection service of the first node is abnormal when the detection result of the first local IO flow detection is abnormal, the detection result of the first peer-end IO flow detection is abnormal, and the detection result of the second local IO flow detection is normal; determine that the IO flow detection service of the second node is abnormal when the detection result of the second local IO flow detection is abnormal, the detection result of the second peer-end IO flow detection is abnormal, and the detection result of the first local IO flow detection is normal.

[0125] The server provided in the embodiment of the present application can execute the technical solution shown in the above method embodiment. Its implementation principles and beneficial effects are similar and will not be repeated here.

[0126] In a possible implementation scheme, the cluster service system is in a hot standby scenario, the first node is the master node, and the second node is the standby node; the processing module 62 is specifically used to switch the cluster service from the first node to the second node for operation when it is determined that the RAID card status of the first node is abnormal and the RAID card status of the second node is normal, and perform alarm processing of the soft fault of the RAID card of the first node; when it is determined that the RAID card status of the second node is abnormal and the RAID card status of the first node is normal, perform alarm processing of the soft fault of the RAID card of the second node; when it is determined that the RAID card status of the first node is abnormal and the RAID card status of the second node is abnormal, perform alarm processing of the soft fault of the RAID cards of the first node and the second node; when it is determined that the IO flow detection service of the first node and / or the second node is abnormal, perform alarm processing of the IO flow detection failure of the first node and / or the second node.

[0127] In a possible implementation scheme, the cluster service system is in a dual-active scenario, the first node and the second node are backup nodes for each other, the first cluster service runs on the first node, and the second cluster service runs on the second node; the processing module 62 is specifically used to switch the first cluster service from the first node to the second node for operation when it is determined that the RAID card status of the first node is abnormal and the RAID card status of the second node is normal, and perform alarm processing of the soft fault of the RAID card of the first node; when it is determined that the RAID card status of the second node is abnormal and the RAID card status of the first node is normal, switch the second cluster service from the second node to the first node for operation, and perform alarm processing of the soft fault of the RAID card of the second node; when it is determined that the RAID card status of the first node is abnormal and the RAID card status of the second node is abnormal, perform alarm processing of the soft fault of the RAID cards of the first node and the second node; when it is determined that the IO flow detection service of the first node and / or the second node is abnormal, perform alarm processing of the IO flow detection failure of the first node and / or the second node.

[0128] The server provided in the embodiment of the present application can execute the technical solution shown in the above method embodiment. Its implementation principles and beneficial effects are similar and will not be repeated here.

[0129] Figure 9 is a schematic diagram of the structure of a server provided in an embodiment of the present application. As shown in Figure 9, the server 70 includes: a processor 71, a memory 72, and a communication interface 73; wherein the memory 72 is used to store executable instructions of the processor 71; the processor 71 is configured to execute the technical solution of any of the aforementioned method embodiments by executing the executable instructions.

[0130] Optionally, the memory 72 can be independent or integrated with the processor 71.

[0131] Optionally, when the memory 72 is a device independent of the processor 71 , the server 70 may further include: a bus 74 for connecting the above devices.

[0132] The server is used to execute the technical solution in any of the aforementioned method embodiments, and its implementation principles and technical effects are similar and will not be repeated here.

[0133] The present application also provides a cluster service system comprising at least one first node and at least one second node, wherein the first node is a master node and the second node is a backup node; wherein the first node executes the technical solution of any of the aforementioned method embodiments.

[0134] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A cluster service processing method, characterized in that: The method is applied to a cluster service system, the cluster service system comprising: a first node and a second node; the method comprises: When the heartbeat network of the first node and the heartbeat network of the second node are normal, obtaining a detection result of a local input and output IO flow detection; When the detection result of the local IO flow detection is abnormal, the detection result of the peer IO flow detection is obtained; Determine the cause of the fault based on the detection result of the peer IO flow detection and the detection result of the local IO flow detection, and process the cluster service according to the cause of the fault; The local input / output IO flow detection is a first local IO flow detection of an IO flow between a first independent redundant disk array RAID card in the first node and a first disk by the first node, and a second local IO flow detection of an IO flow between a second RAID card in the second node and a second disk by the second node by the second node; The peer IO flow detection is a first peer IO flow detection of the IO flow between the second RAID card in the second node and the second disk by the first node, and / or a second peer IO flow detection of the IO flow between the first RAID card in the first node and the first disk by the second node.

2. The cluster service processing method according to claim 1, characterized in that: Get the test results of local IO flow detection, including: The first node and the second node respectively initiate a first data read instruction to the disk of the local node; wherein the first data read instruction is used to read the first data in the disk in the local node; Acquire first data returned by the disk of the local node based on the first data read instruction; The detection result of the local IO flow detection is abnormal, including: the first data returned by the disk of the local node is not obtained.

3. The cluster service processing method according to claim 2, characterized in that: The obtaining the first data returned by the disk of the local node based on the first read data instruction includes: Acquire, within a preset first time period, first data returned by the disk of the local node based on the first data read instruction at preset time intervals; The detection result of the local IO flow detection is abnormal, and also includes: if within the preset first time period, according to the preset time interval point, the total number of times the disk of the local node returns the first data is obtained is less than the first number threshold, then the detection result of the local IO flow detection is determined to be abnormal.

4. The cluster service processing method according to claim 2, characterized in that: The obtaining the first data returned by the disk of the local node based on the first read data instruction includes: Within a preset second time period, obtaining first data returned by the disk of the local node based on the first data read instruction; The detection result of the local IO flow detection is abnormal, further comprising: if the difference between the acquisition time of the first data returned by the disk of the local node and the preset second time is greater than the first time threshold, determining that the detection result of the local IO flow detection is abnormal; The preset second time is the maximum time for the disk of the local node to return the first data normally.

5. The cluster service processing method according to claim 2, characterized in that: The obtaining the first data returned by the disk of the local node based on the first read data instruction includes: Within a preset first time period, obtaining first data returned by the disk of the local node based on the first data read instruction; The detection result of the local IO flow detection is abnormal, and also includes: within the preset first time period, the number of times the difference between the acquisition time of the first data returned by the disk of the local node and the preset second time is greater than the first time threshold is greater than the second number threshold; the preset second time is the maximum time for the first data to be returned by the disk of the local node to be normally acquired.

6. The cluster service processing method according to claim 1, characterized in that: The obtaining of the detection result of the peer IO flow detection includes: The first node or the second node initiates a second data read instruction to the disk of the opposite node; wherein the second data read instruction is used to read second data in the disk of the opposite node; Obtain the second data returned by the disk of the opposite node based on the second data read instruction; if the second data returned by the disk of the opposite node is obtained, determine that the detection result of the opposite IO flow detection is normal; if the second data returned by the disk of the opposite node is not obtained, determine that the detection result of the opposite IO flow detection is abnormal.

7. The cluster service processing method according to claim 1, characterized in that: The determining the cause of the fault based on the detection result of the peer IO flow detection and the detection result of the local IO flow detection includes: When the detection result of the first local IO flow detection is abnormal and the detection result of the first peer IO flow detection is normal, determining that the RAID state of the first node is abnormal; When the detection result of the second local IO flow detection is abnormal and the detection result of the second peer IO flow detection is normal, determining that the RAID state of the second node is abnormal; When the detection result of the first local IO flow detection is abnormal, the detection result of the first peer IO flow detection is abnormal, and the detection result of the second local IO flow detection is normal, it is determined that the IO flow detection service of the first node is abnormal; When the detection result of the second local IO flow detection is abnormal, the detection result of the second peer IO flow detection is abnormal, and the detection result of the first local IO flow detection is normal, it is determined that the IO flow detection service of the second node is abnormal.

8. The cluster service processing method according to claim 7, characterized in that: The processing of the cluster service according to the fault cause includes: The cluster service system is in a hot standby scenario, the first node is a master node, and the second node is a standby node; When it is determined that the RAID card state of the first node is abnormal and the RAID card state of the second node is normal, the cluster service is switched from the first node to the second node for operation, and an alarm processing of a soft fault of the RAID card of the first node is performed; When it is determined that the RAID card state of the second node is abnormal and the RAID card state of the first node is normal, performing alarm processing of a soft fault of the RAID card of the second node; When it is determined that the RAID card state of the first node is abnormal and the RAID card state of the second node is abnormal, an alarm process of a soft failure of the RAID cards of the first node and the second node is performed; When it is determined that the IO flow detection service of the first node and / or the second node is abnormal, an alarm process of the IO flow detection failure of the first node and / or the second node is performed.

9. The cluster service processing method according to claim 7, characterized in that: The processing of the cluster service according to the fault cause includes: The cluster service system is in an active-active scenario, the first node and the second node are each other's standby nodes, a first cluster service runs on the first node, and a second cluster service runs on the second node; When it is determined that the RAID card state of the first node is abnormal and the RAID card state of the second node is normal, the first cluster service is switched from the first node to the second node for operation, and an alarm processing of a soft fault of the RAID card of the first node is performed; When it is determined that the RAID card state of the second node is abnormal and the RAID card state of the first node is normal, switching the second cluster service from the second node to the first node for operation, and performing alarm processing for a soft fault of the RAID card of the second node; When it is determined that the RAID card state of the first node is abnormal and the RAID card state of the second node is abnormal, an alarm process of a soft failure of the RAID cards of the first node and the second node is performed; When it is determined that the IO flow detection service of the first node and / or the second node is abnormal, an alarm process of the IO flow detection failure of the first node and / or the second node is performed.

10. A cluster service system, characterized in that: include: At least one first node and at least one second node, wherein the first node is a primary node and the second node is a backup node; Wherein, the first node executes the cluster service processing method according to any one of claims 1 to 9.

Citation Information

Patent Citations

  • Data processing method and system

    CN107247564A

  • Fault processing method and device for database server

    CN115686951A

  • Cluster service processing method, server and system

    CN116668335A

  • Method and device for providing redundancy to raid card

    JP2011076528A

  • Failure detection and failure handling in cluster controller networks

    US20030182592A1

Cited By

  • Abnormal reason detection method and device, equipment, storage medium and program product

    CN120896834A