A cluster service processing method, server and system

Abstract

The application provides a cluster service processing method, a server and a system. The method is applied to a cluster service system, and the cluster service system comprises a first node and a second node. The method comprises the following steps: acquiring a detection result of local input / output (IO) flow detection in the case that a heartbeat network of the first node and a heartbeat network of the second node are normal; acquiring a detection result of opposite-end IO flow detection in the case that the detection result of the local IO flow detection is abnormal; determining a fault cause based on the detection result of the opposite-end IO flow detection and the detection result of the local IO flow detection, and processing the cluster service according to the fault cause. The method solves the problem that the cluster service system cannot switch the service from the node in which the RAID card fails to the normal node for running due to the fact that the heartbeat network detection cannot detect the RAID card failure in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of database, and particularly relates to a cluster service processing method, a server and a system. BACKGROUND

[0002] With the rapid development of digitization and informatization, a large amount of data is generated, and the existing technology usually stores and analyzes the data in the form of a database. In order to ensure the continuity of the service, a high-availability cluster is usually used to build a database. When the high-availability cluster detects that one or more nodes in the cluster fail, the service is switched from the failed node to the normal node, thereby avoiding the interruption of the service.

[0003] In the prior art, the node switching in the high-availability cluster usually depends on heartbeat network detection. The heartbeat network detection mainly detects whether the node fails by listening to the heartbeat signal of the node in the cluster. When the heartbeat signal of a node in the cluster is not listened to within a specified time, it is determined that the node fails, and the service running on the node is switched to a normal node. However, in the case where the node does not fail, if the redundant array of independent disks (RAID) card for controlling data storage in the node fails, the IO stream of the multiple disks managed by the RAID card cannot be normally read and written, thereby affecting the database service. At this time, node switching is also needed. However, since the node does not fail, the heartbeat signal of the node can still be listened to, so the cluster service system does not switch the service from the node where the RAID card fails to a normal node. SUMMARY

[0004] The embodiments of the present application provide a cluster service processing method, a server and a system, which are used to solve the problem that, in the prior art, the cluster service system does not switch the service from the node where the RAID card fails to a normal node because the heartbeat network detection cannot detect the failure of the RAID card in the node.

[0005] In a first aspect, an embodiment of the present application provides a cluster service processing method, which is applied to a cluster service system, the cluster service system comprising a first node and a second node; the method comprising: in a case where a heartbeat network of the first node and a heartbeat network of the second node are normal, obtaining a detection result of local input / output (IO) flow detection; in a case where the detection result of the local IO flow detection is abnormal, obtaining a detection result of peer IO flow detection; determining a fault cause based on the detection result of the peer IO flow detection and the detection result of the local IO flow detection, and processing the cluster service according to the fault cause; wherein the local IO flow detection comprises first local IO flow detection of IO flow between a first independent redundant array of inexpensive disks (RAID) card and a first disk in the first node by the first node, and second local IO flow detection of IO flow between a second RAID card and a second disk in the second node by the second node; the peer IO flow detection comprises first peer IO flow detection of IO flow between the second RAID card and the second disk in the second node by the first node, and / or second peer IO flow detection of IO flow between the first RAID card and the first disk in the first node by the second node.

[0006] In an embodiment of the present application, by introducing an IO flow detection service, each node detects local IO flow, and in a case where the detection result is abnormal, each node detects peer IO flow. Since the local IO flow detection service is abnormal, it may be caused by abnormal local IO flow detection service or soft failure of a local RAID card. Therefore, in a case where the local IO flow detection is abnormal, by detecting the peer IO flow, it can be further determined whether the soft failure of the local RAID card occurs, thereby solving the problem that the system is dead and the cluster service system master and backup node switching is not performed when the disk cannot be normally read and written, and improving the reliability of the cluster service system.

[0007] In a specific embodiment, obtaining the detection result of the local IO flow detection comprises: the first node and the second node respectively initiate a first read data instruction to a disk of a local node; wherein the first read data instruction is used to read first data in the disk of the local node; obtaining first data returned by the disk of the local node based on the first read data instruction; and the detection result of the local IO flow detection is abnormal, comprising: no first data returned by the disk of the local node is obtained.

[0008] In the above embodiment, the first node and the second node respectively send a read data instruction to the local disk of each node, and return data based on the data read instruction. If the data in the local disk can be obtained, the local IO stream detection result is normal. If the data in the local disk cannot be obtained, the local IO stream detection result is abnormal. Through the above method, the detection of the local IO stream of each node is realized.

[0009] In a specific embodiment, the first data returned by the disk of the local node based on the first read data instruction is obtained by: obtaining the first data returned by the disk of the local node based on the first read data instruction at preset time interval points within a preset first time period. If the sum of the number of times of obtaining the first data returned by the disk of the local node within the preset first time period at the preset time interval points is less than a first number threshold, it is determined that the detection result of the local IO stream detection is abnormal.

[0010] In the above embodiment, each node reads the data in the disk of the local node at time interval points within a preset first time period. If the total number of times of obtaining the first data returned by the disk of the local node within the preset first time period is greater than or equal to a first number threshold, it is determined that the detection result of the local IO stream detection is normal. Otherwise, it is determined that the detection result of the local IO stream detection is abnormal. Through the above method, the number of times of successfully obtaining the first data within a preset time period is determined, so that the detection result of the IO stream detection is more accurate and reliable.

[0011] In a specific embodiment, the first data returned by the disk of the local node based on the first read data instruction is obtained by: obtaining the first data returned by the disk of the local node based on the first read data instruction within a preset second time period. If the difference between the acquisition time of the first data returned by the disk of the local node and the preset second time is greater than a first time threshold, it is determined that the detection result of the local IO stream detection is abnormal. The preset second time is the maximum time of normally obtaining the first data returned by the disk of the local node.

[0012] In the above embodiment, the timeout time of each node for obtaining the first data is determined within a preset second time period. If the timeout time is greater than a first time threshold, it is determined that the detection result of the local IO stream detection is abnormal. Otherwise, it is determined that the detection result of the local IO stream detection is normal. Through the above method, the detection result of the local IO stream detection is more reliable.

[0013] In an embodiment, the acquiring the first data returned by the disk of the local node based on the first read data instruction comprises: acquiring the first data returned by the disk of the local node based on the first read data instruction within a preset first time period; and the detection result of the local IO stream detection is abnormal, further comprising: the difference between the acquisition time of the first data returned by the disk of the local node and a preset second time is greater than a first time threshold value for a number of times greater than a second number threshold value within the preset first time period; and the preset second time is a maximum time for normally acquiring the first data returned by the disk of the local node.

[0014] In the above embodiment, the detection result of the local IO stream detection is determined by judging the number of timeouts of the first node in acquiring the first data within the first time period, which further improves the reliability and accuracy of the detection.

[0015] In an embodiment, the acquiring the first data returned by the disk of the local node based on the first read data instruction comprises: acquiring the first data returned by the disk of the local node based on the first read data instruction within a preset first time period; and the detection result of the local IO stream detection is abnormal, further comprising: the difference between the acquisition time of the first data returned by the disk of the local node and a preset second time is greater than a first time threshold value for a number of times greater than a second number threshold value within the preset first time period; and the preset second time is a maximum time for normally acquiring the first data returned by the disk of the local node.

[0016] In the above embodiment, each node sends a read data instruction to the disk of the opposite node, and if the second data returned by the opposite node can be acquired, it is determined that the detection result of the opposite IO stream is normal, otherwise, it is abnormal. By increasing the detection of the opposite IO stream, the detection result of the local IO stream can be further verified, and the accuracy and reliability of the fault judgment are enhanced.

[0017] In a specific embodiment, the determining of the fault cause based on the detection result of the peer IO stream detection and the detection result of the local IO stream detection comprises: in a case that the detection result of the first local IO stream detection is abnormal and the detection result of the first peer IO stream detection is normal, determining that the RAID state of the first node is abnormal; in a case that the detection result of the second local IO stream detection is abnormal and the detection result of the second peer IO stream detection is normal, determining that the RAID state of the second node is abnormal; in a case that the detection result of the first local IO stream detection is abnormal, the detection result of the first peer IO stream detection is abnormal, and the detection result of the second local IO stream detection is normal, determining that the IO stream detection service of the first node is abnormal; in a case that the detection result of the second local IO stream detection is abnormal, the detection result of the second peer IO stream detection is abnormal, and the detection result of the first local IO stream detection is normal, determining that the IO stream detection service of the second node is abnormal.

[0018] In the above embodiment, in a case that the detection result of the local IO stream detection is abnormal, the detection result of the peer IO stream detection is obtained, and the fault cause is determined based on the detection result of the local IO stream detection and the detection result of the peer IO stream detection, so that the fault of the cluster service system can be accurately determined.

[0019] In a specific embodiment, the method further comprises: in a case that the heartbeat detection result of the first node is abnormal and the heartbeat detection result of the second node is normal, determining that the heartbeat network of the first node is abnormal; in a case that the heartbeat detection result of the first node is normal and the heartbeat detection result of the second node is abnormal, determining that the heartbeat network of the second node is abnormal; in a case that the heartbeat detection result of the first node and the heartbeat detection result of the second node are both abnormal, determining that the heartbeat network of the first node and the heartbeat network of the second node are both abnormal.

[0020] In the above embodiment, in addition to the system hang condition of each node, the cluster service system can also have other faults, the heartbeat network of each node is detected, and whether the heartbeat network of each node is normal is determined based on the detection result, so that the fault of the cluster service system can be more accurately determined.

[0021] In an embodiment, the processing of the cluster service according to the fault cause comprises: when the cluster service system is in a hot standby scenario, the first node is a master node, and the second node is a standby node; when it is determined that the RAID card of the first node is abnormal and the RAID card of the second node is normal, switching the cluster service from the first node to the second node, and performing alarm processing of the soft fault of the RAID card of the first node; when it is determined that the RAID card of the second node is abnormal and the RAID card of the first node is normal, performing alarm processing of the soft fault of the RAID card of the second node; when it is determined that the RAID card of the first node is abnormal and the RAID card of the second node is abnormal, performing alarm processing of the soft fault of the RAID card of the first node and the second node; when it is determined that the IO stream detection service of the first node and / or the second node is abnormal, performing alarm processing of the IO stream detection fault of the first node and / or the second node.

[0022] In the above embodiment, for the system in a hot standby scenario, the cluster service system is processed according to each fault cause, thereby improving the reliability of system operation.

[0023] In an embodiment, the processing of the cluster service according to the fault cause comprises: when the cluster service system is in a hot standby scenario, the first node is a master node, and the second node is a standby node; when it is determined that the RAID card of the first node is abnormal and the RAID card of the second node is normal, switching the cluster service from the first node to the second node, and performing alarm processing of the soft fault of the RAID card of the first node; when it is determined that the RAID card of the second node is abnormal and the RAID card of the first node is normal, switching the cluster service from the first node to the second node, and performing alarm processing of the soft fault of the RAID card of the first node; when it is determined that the RAID card of the first node is abnormal and the RAID card of the second node is abnormal, performing alarm processing of the soft fault of the RAID card of the first node and the second node; when it is determined that the IO stream detection service of the first node and / or the second node is abnormal, performing alarm processing of the IO stream detection fault of the first node and / or the second node.

[0024] In the above embodiment, for the system in a hot standby scenario, the cluster service system is processed according to each fault cause, thereby improving the reliability of system operation.

[0025] In a second aspect, an embodiment of the present application provides a server, comprising: a processor, a memory, and a communication interface; the memory is configured to store executable instructions of the processor; wherein the processor is configured to execute the cluster service processing method according to the first aspect by executing the executable instructions.

[0026] In a third aspect, an embodiment of the present application provides a cluster service system, comprising: at least one first node and at least one second node, the first node is a master node, and the second node is a standby node; wherein the first node executes the cluster service processing method according to the first aspect.

[0027] The embodiments of the present application provide a cluster service processing method, a server and a system. The method is applied to a cluster service system, and the cluster service system comprises a first node and a second node. The method comprises: obtaining a detection result of local input / output (IO) flow detection in a case where a heartbeat network of the first node and a heartbeat network of the second node are normal; obtaining a detection result of opposite-end IO flow detection in a case where the detection result of the local IO flow detection is abnormal; determining a fault cause based on the detection result of the opposite-end IO flow detection and the detection result of the local IO flow detection, and processing the cluster service according to the fault cause; wherein the local IO flow detection comprises first local IO flow detection of IO flow between a first redundant array of independent disks (RAID) card and a first disk in the first node by the first node, and second local IO flow detection of IO flow between a second RAID card and a second disk in the second node by the second node; and the opposite-end IO flow detection comprises first opposite-end IO flow detection of IO flow between the second RAID card and the second disk in the second node by the first node, and / or second opposite-end IO flow detection of IO flow between the first RAID card and the first disk in the first node by the second node. Compared with the prior art which relies on heartbeat network detection to realize switching of services from a fault node to a normal node, the embodiments of the present application determine a fault cause of a cluster service according to local IO flow detection and opposite-end IO flow detection of the first node and the second node, in combination with heartbeat detection results of the first node and the second node, process the cluster service according to the fault cause, and enable the cluster service to switch services from a fault node to a normal node in time when the services need to be switched due to a RAID card soft fault, thereby solving the problem in the prior art that a cluster service system cannot switch services from a node in which a RAID card has a fault to a normal node for running due to the fact that the heartbeat network detection cannot detect the RAID card soft fault in the node. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced. Obviously, the accompanying drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0029] Figure 1 The structure diagram of the cluster service system;

[0030] Figure 2 The flowchart of the cluster service processing method provided by the embodiment of the present application is shown in Figure 1.

[0031] Figure 3 The schematic diagram of the IO flow detection of the cluster service system is shown in Figure 2.

[0032] Figure 4 The flowchart of the cluster service processing method provided by the embodiment of the present application is shown in Figure 1.

[0033] Figure 5 The flowchart of the cluster service processing method provided by the embodiment of the present application is shown in Figure 1.

[0034] Figure 6 The flowchart of the cluster service processing method provided by the embodiment of the present application is shown in Figure 1.

[0035] Figure 7 The flowchart of the cluster service processing method provided by the embodiment of the present application is shown in Figure 1.

[0036] Figure 8 The structure diagram of the server provided by the embodiment of the present application is shown in Figure 3.

[0037] Figure 9 The structure diagram of another server provided by the embodiment of the present application is shown in Figure 4. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be described clearly and completely with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments made by those skilled in the art according to the inspiration of the embodiments of the present application are within the scope of protection of the present application.

[0039] The terms "first", "second", "third", "fourth" and the like in the description and in the claims of the present application and above-mentioned drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of these terms herein is merely for distinguishing between the objects and not necessarily for describing a particular sequential or chronological order, unless explicitly stated otherwise. The terms "comprising," "having," "including," and "containing" and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, product, or apparatus that comprises, has, includes or contains an item or list of items who does not also preclude any other

[0040] First, the terms involved in the present application are explained:

[0041] RAID card: Redundant Array of Independent Disks (RAID) combines multiple independent disks into a large-capacity disk group. The RAID card is to manage multiple disks that make up the disk array. When implementing database services, the operating system needs to read and write data to the disks managed by the RAID card.

[0042] Figure 1 The structure of the cluster service system is shown in the figure. The cluster service system includes a first node 11 and a second node 12. Among them, the first node 11 is the master node, and the second node 12 is the standby node. The first node 11 and the second node 12 perform heartbeat network detection through the heartbeat network link 13. Heartbeat network detection mainly detects whether the node in the cluster fails by listening to the heartbeat signal of the node. Specifically, the first node 11 and the second node 12 send and receive heartbeat messages to each other through the heartbeat network link 13 at a fixed frequency. If the second node 12 does not receive the heartbeat message of the first node 11 within a specified time, it is determined that the first node 11 has failed, and the service running on the first node 11 is switched to the second node 12.

[0043] As Figure 1As shown, the first node 11 and the second node 12 each include a processor running an operating system, a RAID card, and a plurality of disks managed by the RAID card. To implement database services, the operating system needs to read and write data to the disks managed by the RAID card. If the RAID card for controlling data reading and writing in the node fails, for example, the RAID card program runs an error, the IO stream of the plurality of disks managed by the RAID card cannot be normally read and written, and thus the database services are affected, at which time node switching needs to be performed. Since the node does not fail, the heartbeat signal can still be monitored, and thus the cluster service system does not switch the services from the node with the failed RAID card to the normal node.

[0044] Based on the above technical problem, the technical concept of the present application is as follows: how to detect the RAID card failure of the node to switch the cluster services from the node with the failed RAID card to the normal node.

[0045] In the following, the technical solutions of the present application are described in detail through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described in detail in some embodiments.

[0046] Figure 2 A flowchart of a cluster service processing method embodiment one provided by the present application. The method is applied to a cluster service system, and the cluster service system includes a first node and a second node. Referring to Figure 2 The cluster service processing method specifically includes the following steps:

[0047] Step S201: In the case that the heartbeat network of the first node and the heartbeat network of the second node are normal, obtaining a detection result of local input and output IO stream detection.

[0048] In an example, the cluster service system is in a hot standby scene, including a first node and a second node, wherein the first node is a master node, and the second node is a standby node. Database services run on the first node, and when the first node fails, the database services are switched from the first node to the second node to run. The cluster service system can also include a management node. The first node and the second node are externally connected to a storage device. Figure 3 A schematic diagram of IO stream detection for a cluster service system.

[0049] Among them, the local IO stream detection is a first local IO stream detection of the first node on the IO stream between the first independent redundant disk array RAID card in the first node and the first disk, and / or a second local IO stream detection of the second node on the IO stream between the second RAID card in the second node and the second disk. As Figure 3As shown, the first local IO flow detection is the detection of the IO flow between the first RAID card and the first disk in the first node; and the second local IO flow detection is the detection of the IO flow between the second RAID card and the second disk in the second node. The local IO flow detection is mainly used to detect whether the local system disk has a problem of being unable to normally read and write.

[0050] In the database service cluster, the first node is the master node, and the second node is the standby node. The first node and the second node each include a RAID card and a plurality of disks managed by the RAID card. The disks can be system disks. To implement the database service, the operating system needs to read and write data on the disks managed by the RAID card. In the process of reading and writing data, the IO flow is formed between the RAID card and the disks. Therefore, whether the RAID card has a program running error or other soft failure can be determined by detecting the IO flow between the RAID card and the disks. When the detection result of the IO flow between the RAID card and the disks is abnormal, it can be determined that the RAID card has a program running error or other soft failure.

[0051] In this embodiment, the detection result of the local input / output (IO) flow detection is obtained when the heartbeat network of the first node and the heartbeat network of the second node are normal. Specifically, the first node detects the IO flow between the first RAID card and the first disk in the first node to obtain the detection result of the first local IO flow detection; and the second node detects the IO flow between the second RAID card and the second disk in the second node to obtain the detection result of the second local IO flow detection. The first node as the master node obtains the detection results of the first local IO flow detection and the second local IO flow detection, or the management node obtains the detection results of the first local IO flow detection and the second local IO flow detection.

[0052] Step S202: When the detection result of the local IO flow detection is abnormal, the detection result of the peer IO flow detection is obtained.

[0053] The peer IO flow detection is the first peer IO flow detection of the first node on the IO flow between the second RAID card and the second disk in the second node, and / or the second peer IO flow detection of the second node on the IO flow between the first RAID card and the first disk in the first node. As shown, the first peer IO flow detection is the detection of the IO flow between the second RAID card and the second disk in the second node by the first node; and the second peer IO flow detection is the detection of the IO flow between the first RAID card and the first disk in the first node by the second node. The peer IO flow detection is used to check the result, ensure the consistency and reliability of the result, and check whether the IO flow detection service itself is abnormal. Figure 3 The peer IO flow detection is the first peer IO flow detection of the first node on the IO flow between the second RAID card and the second disk in the second node, and / or the second peer IO flow detection of the second node on the IO flow between the first RAID card and the first disk in the first node. As shown, the first peer IO flow detection is the detection of the IO flow between the second RAID card and the second disk in the second node by the first node; and the second peer IO flow detection is the detection of the IO flow between the first RAID card and the first disk in the first node by the second node. The peer IO flow detection is used to check the result, ensure the consistency and reliability of the result, and check whether the IO flow detection service itself is abnormal.

[0054] In the embodiment, when the detection result of the local IO flow detection is abnormal, the detection result of the peer IO flow detection is obtained. Specifically, the first node detects the IO flow between the second RAID card and the second disk in the second node to obtain the detection result of the first peer IO flow detection; the second node also detects the IO flow between the first RAID card and the first disk in the first node to obtain the detection result of the second peer IO flow detection. The first node as the master node obtains the detection results of the first and second peer IO flow detections, and the management node can also obtain the detection results of the first and second peer IO flow detections.

[0055] In the embodiment, the first node and the second node also perform heartbeat detection. The heartbeat packets are sent and received at a fixed frequency to obtain the heartbeat detection result. The first node as the master node obtains the heartbeat detection result of the first node and the heartbeat detection result of the second node, and the management node can also obtain the heartbeat detection result of the first node and the heartbeat detection result of the second node. When the heartbeat network of the first node and the heartbeat network of the second node are normal, the steps S201 to S202 are executed.

[0056] Step S203: Based on the detection result of the peer IO flow detection and the detection result of the local IO flow detection, the fault cause is determined, and the cluster service is processed according to the fault cause.

[0057] In the embodiment, the first node or the management node determines the fault cause based on the detection result of the peer IO flow detection and the detection result of the local IO flow detection.

[0058] For example, when the heartbeat detection results of the first node and the second node are normal, the detection result of the first local IO flow detection is abnormal, and the detection result of the first peer IO flow detection is normal, it can be initially determined that the RAID card state of the first node is abnormal, and when the detection result of the second local IO flow detection is normal, it indicates that the IO detection service of the second node is normal, and when the detection result of the second peer IO flow detection is abnormal, it can be further determined that the RAID state of the first node is abnormal and the RAID state of the second node is normal.

[0059] When the first node is the master node and the second node is the standby node, the RAID state of the first node is abnormal and the RAID state of the second node is normal, the cluster service needs to be switched, and the cluster service running on the first node is switched to run on the second node.

[0060] In an example, the cluster service system is in a dual-active scenario, including a first node and a second node, the first node and the second node being standby nodes of each other. The database service A is running on the first node, and the database service B is running on the second node. When the first node fails, the database service A is switched from the first node to the second node to run; when the second node fails, the database service B is switched from the second node to the first node to run. The cluster service system can further include a management node.

[0061] In a case where the heartbeat network of the first node and the heartbeat network of the second node are normal, a detection result of local input / output (IO) flow detection can be obtained by the first node, the second node, or the management node. In a case where the detection result of the local IO flow detection is abnormal, a detection result of peer IO flow detection is obtained. The first node, the second node, or the management node determines a failure cause based on the detection result of the peer IO flow detection and the detection result of the local IO flow detection, and processes the cluster service according to the failure cause.

[0062] In an example, the detection results of the first local IO flow detection and the first peer IO flow detection, the detection results of the second local IO flow detection and the second peer IO flow detection, and the heartbeat detection results of the first node and the second node can be obtained when the database service is started. The RAID card states of the first node and the second node are determined based on the detection results of the first local IO flow detection, the first peer IO flow detection, the second local IO flow detection, and the second peer IO flow detection, and the heartbeat detection results of the first node and the second node, respectively, to determine whether to switch the cluster service.

[0063] In an example, the detection results of the first local IO flow detection and the first peer IO flow detection, the detection results of the second local IO flow detection and the second peer IO flow detection, and the heartbeat detection results of the first node and the second node can be obtained every preset time after the database service is started. The RAID card states of the first node and the second node are determined based on the detection results of the first local IO flow detection, the first peer IO flow detection, the second local IO flow detection, and the second peer IO flow detection, and the heartbeat detection results of the first node and the second node, respectively, to determine whether to switch the cluster service.

[0064] In an example, the detection result of the first local IO flow detection and the first peer IO flow detection, the detection result of the second local IO flow detection and the second peer IO flow detection, and the heartbeat detection result of the first node and the second node can also be acquired multiple times within a preset time. After the RAID card states of the first node and the second node are determined according to the multiple detection results of the first local IO flow detection, the first peer IO flow detection, the second local IO flow detection and the second peer IO flow detection, and the multiple heartbeat detection results of the first node and the second node, the number of times of abnormal RAID card states of the first node and the second node is counted, and whether to perform the switching of the cluster service is determined according to the number of times of abnormal RAID card states of the first node and the second node. Exemplarily, the first node is a master node, and the second node is a standby node. When the number of times of abnormal RAID card states of the first node exceeds a threshold value, and the number of times of abnormal RAID card states of the second node is 0, that is, the multiple detection results of the RAID card of the second node are all normal, the switching of the cluster service is performed.

[0065] In the embodiment, the cluster service system includes a first node and a second node. When the heartbeat network of the first node and the heartbeat network of the second node are normal, a detection result of a local input and output (IO) flow detection is acquired. When the detection result of the local IO flow detection is abnormal, a detection result of a peer IO flow detection is acquired. Based on the detection result of the peer IO flow detection and the detection result of the local IO flow detection, a fault reason is determined, and the cluster service is processed according to the fault reason. The local IO flow detection is a first local IO flow detection of IO flow between a first redundant array of independent disks (RAID) card and a first disk in the first node by the first node, and / or a second local IO flow detection of IO flow between a second RAID card and a second disk in the second node by the second node. The peer IO flow detection is a first peer IO flow detection of IO flow between the second RAID card and the second disk in the second node by the first node, and / or a second peer IO flow detection of IO flow between the first RAID card and the first disk in the first node by the second node. Compared with the prior art which relies on heartbeat network detection to realize the switching of the service from a fault node to a normal node, the cluster service of the application can be switched from a fault node to a normal node in time when the service needs to be switched due to a RAID card soft fault, and the problem that the cluster service system cannot switch the service from a node with a fault RAID card to a normal node in the prior art is solved.

[0066] Figure 4 This is a flowchart illustrating a second embodiment of a cluster service processing method provided in this application. Figure 2 Based on the embodiment shown, step S201 may include: the first node and the second node respectively sending a first read data instruction to the disk of the local node; wherein, the first read data instruction is used to read first data in the disk of the local node; obtaining the first data returned by the disk of the local node based on the first read data instruction; the detection result of the local IO stream detection is abnormal, including: not obtaining the first data returned by the disk of the local node.

[0067] Specifically, step S201 may include the following steps:

[0068] Step S401: The first node initiates a first read data instruction to the first disk in the first node; the first read data instruction is used to read the first data in the first disk.

[0069] Step S402: The first node obtains the first data returned by the first disk based on the first read data instruction; if the first data returned by the first disk is obtained, the detection result of the first local IO stream detection is determined to be normal; if the first data returned by the first disk is not obtained, the detection result of the first local IO stream detection is determined to be abnormal.

[0070] In this embodiment, the RAID card manages multiple disks. When implementing database services, the operating system needs to read and write data to the disks managed by the RAID card. During the data read and write process, an I / O flow is formed between the RAID card and the disks. Therefore, by detecting the I / O flow between the RAID card and the disks, it can be determined whether a soft failure has occurred in the RAID card. When the RAID card experiences a soft failure such as a program runtime error, the disk will not respond to read commands and return data. Therefore, the I / O flow between the RAID card and the system disk can be detected by sending read data commands to the disk, thereby determining whether a soft failure has occurred in the RAID card. For example, underlying I / O testing tools, such as a disk stress tester (Flexible Input Output tester, FIO) or I / O testing software (Input Output meter, IOmeter), can be called to perform read operations on the disk.

[0071] Specifically, when performing the first local IO flow detection, the first node initiates a first read data instruction to a first disk in the first node, the first read data instruction being used to read first data in the first disk managed by a first RAID card, and if the RAID card in the first node is in a normal state, the first disk returns the first data to an operating system of the first node based on the first read data instruction; if the RAID card in the first node has a soft failure, the first disk does not return the data.

[0072] The first node acquires the first data returned by the first disk based on the first read data instruction, and if the first data returned by the first disk is successfully acquired, it is determined that a detection result of the first local IO flow detection is normal; if the first data returned by the first disk is not acquired, it is determined that the detection result of the first local IO flow detection is abnormal.

[0073] On the basis of the embodiments shown above, Figure 2 On the basis of the embodiments shown above, the step S202 can include that the first node or the second node initiates a second read data instruction to a disk in a peer node; the second read data instruction is used to read second data in the disk in the peer node; second data returned by the disk in the peer node based on the second read data instruction is acquired; if the second data returned by the disk in the peer node is acquired, it is determined that a detection result of the peer IO flow detection is normal; and if the second data returned by the disk in the peer node is not acquired, it is determined that the detection result of the peer IO flow detection is abnormal.

[0074] Specifically, the step S202 can include the following steps:

[0075] The step S403 includes that the first node initiates a second read data instruction to a second disk in the second node; the second read data instruction is used to read second data in the second disk.

[0076] The step S404 includes that the first node acquires second data returned by the second disk based on the second read data instruction; if the second data returned by the second disk is acquired, it is determined that a detection result of the first peer IO flow detection is normal; and if the second data returned by the second disk is not acquired, it is determined that the detection result of the first peer IO flow detection is abnormal.

[0077] It should be noted that the execution sequence of the steps S402 to S404 is not specifically limited.

[0078] In the embodiment, the first node also performs IO stream detection on the second node opposite to the first node, i.e., first opposite node IO stream detection. Specifically, a second read data instruction is initiated to the second disk in the second node, and the second read data instruction is used to read second data in the second disk. If the RAID card in the second node is in a normal state, the second disk returns the second data to the operating system of the first node based on the second read data instruction; if the RAID card in the second node has a soft failure, the second disk does not return data.

[0079] The first node acquires the second data returned by the second disk based on the second read data instruction; if the second data returned by the second disk is successfully acquired, it is determined that the detection result of the first opposite node IO stream detection is normal; if the second data returned by the second disk is not acquired, it is determined that the detection result of the first opposite node IO stream detection is abnormal.

[0080] In the embodiment, the attribute parameters of the IO stream detection include interval time, timeout time, timeout times, etc. For example, the first node can perform local IO stream detection and opposite node IO stream detection every preset interval time. For example, the first node acquires the first data returned by the first disk within a preset first time period at a preset time interval node. If the first data returned by the first disk can be acquired at each time interval node within the preset first time period, it is determined that the detection result of the first local IO stream detection is normal. Or, the first node acquires the first data returned by the first disk at all time interval nodes within the preset first time period, and the number of times is greater than or equal to a first number threshold. It is determined that the detection result of the first local IO stream detection is normal. If the first node cannot acquire the first data returned by the first disk at all time interval nodes within the preset first time period, it is determined that the detection result of the first local IO stream detection is abnormal. Or, the first node acquires the first data returned by the first disk at all time interval nodes within the preset first time period, and the number of times is less than the first number threshold. It is determined that the detection result of the first local IO stream detection is abnormal. The first number threshold is the minimum number of times that the first node can acquire the first data returned by the first disk within the preset first time period. The interval time can be 1s, 5s, 1min, 1h, etc., which is not particularly limited here.

[0081] In some implementations, if the acquisition time of the first data returned by the first disk by the first node is less than a first time threshold from a preset first time, it is determined that the detection result of the first local IO stream detection is normal. If the acquisition time of the first data returned by the first disk by the first node is greater than a first time threshold from a preset second time, it is determined that the detection result of the first local IO stream detection is abnormal, wherein the second time is the maximum time that the first node can normally acquire the first data returned by the first disk.

[0082] In some implementations, when the first node obtains the first data returned by the first disk for a preset first time period and the number of times of timeout is greater than a second number threshold, it is determined that the detection result of the first local IO flow detection is abnormal; when the first node obtains the first data returned by the first disk for the preset first time period and the number of times of timeout is less than or equal to the second number threshold, it is determined that the detection result of the first local IO flow detection is normal. The number of times of timeout is the maximum number of times that the difference between the obtaining time of the first node obtaining the first data returned by the first disk and the preset second time is greater than a first time threshold within the time period.

[0083] Similarly, the second node performs the second local IO flow detection and the second peer IO flow detection, which can be performed according to the steps S401 to S404 described above.

[0084] In this embodiment, the first node performs the first local IO flow detection and the first peer IO flow detection by respectively initiating a read data instruction to the first disk and the second disk in the first node to read data from the first disk and the second disk, so as to respectively detect the running state of the RAID card of the local node and the RAID card of the peer node. Therefore, the IO flow of the local node and the peer node can be detected without affecting the database service, so as to determine whether the RAID has a soft failure, thereby providing a precondition for timely switching the service from the failed node to the normal node when the service needs to be switched due to the soft failure of the RAID card.

[0085] Figure 5 The flowchart of the third embodiment of the cluster service processing method provided in this application is shown in the above Figures 2 to 4 On the basis of the above-described embodiment, the step S203 specifically includes the following steps:

[0086] Step S501: According to the detection results of the first local IO flow detection, the first peer IO flow detection, the second local IO flow detection and the second peer IO flow detection, and the heartbeat detection results of the first node and the second node, the RAID card states of the first node and the second node are respectively determined.

[0087] Among the first node and the second node, one is a master node and the other is a standby node.

[0088] In this embodiment, according to the detection results of the first local IO flow detection, the first peer IO flow detection, the second local IO flow detection and the second peer IO flow detection, and the heartbeat detection results of the first node and the second node, the RAID card states of the first node and the second node are determined.

[0089] Local IO stream detection is mainly used to detect whether there are problems with normal reading and writing of the local system disk; peer IO stream detection is used to verify the local IO stream detection results of the peer node to ensure the consistency and reliability of the results, and to verify whether there are any abnormalities in the IO stream detection service itself.

[0090] Specifically, if the heartbeat detection results of both the first and second nodes are normal, the detection result of the first local IO stream is abnormal, and the detection result of the first peer IO stream is normal, then the RAID status of the first node is determined to be abnormal. Normal heartbeat detection results of the first and second nodes indicate that the network between the first and second nodes is normal; an abnormal detection result of the first local IO stream indicates that the first node's local disk may have a problem with normal read / write operations (or the first node's IO stream detection service may be abnormal). To further verify whether the first node's local disk has a problem with normal read / write operations, it is necessary to judge the first peer IO stream detection result. If the first peer IO stream detection result is normal, the possibility of an abnormal first node IO stream detection service can be ruled out. Therefore, it can be determined that the RAID status of the first node is abnormal.

[0091] If the heartbeat detection results of the first node and the second node are both normal, and the detection result of the first local IO stream is normal, it can be preliminarily determined that the RAID card of the first node and the IO detection service of the first node are normal. In order to further verify the above conclusion, the result of the second peer IO stream detection can be used for judgment. If the detection result of the second peer IO stream detection is normal, it is determined that the RAID status of the first node is normal.

[0092] Similarly, if the heartbeat detection results of both the first and second nodes are normal, and the first local IO stream detection and the second peer IO stream detection are normal, then if the detection result of the first peer IO stream is abnormal, it can be preliminarily determined that the RAID card of the second node is abnormal. If the detection result of the second local IO stream is also abnormal, then it is further confirmed that the RAID status of the second node is abnormal. If the heartbeat detection results of both the first and second nodes are normal, it indicates that the network between the first and second nodes is normal. If the first local IO stream detection and the second peer IO stream detection are both normal, it indicates that the IO stream detection services of both the first and second nodes are normal. If the detection result of the second local IO stream is abnormal, it can be preliminarily determined that the local disk of the second node has a problem with normal read / write operations. To further verify this conclusion, the detection result of the first peer IO stream is used for judgment. If the detection result is abnormal, then the RAID card status of the second node is abnormal, further verifying that the local disk of the second node has a problem with normal read / write operations. Therefore, it can be further confirmed that the RAID status of the second node is abnormal.

[0093] When the heartbeat detection results of the first node and the second node are both normal, and the detection result of the second local IO stream detection is normal, it can be initially determined that the local disk read and write of the second node are normal. In order to further verify the above conclusion, the detection result of the first peer IO stream detection is verified. If the result is normal, it is determined that the RAID state of the second node is normal.

[0094] Specifically, when the heartbeat detection results of the first node and the second node are both normal, the detection result of the first local IO stream detection is abnormal, the detection result of the first peer IO stream detection is abnormal, the detection result of the second local IO stream detection is normal, and the detection result of the second peer IO stream detection is normal, it is determined that the IO stream detection service of the first node is abnormal. The heartbeat detection results of the first node and the second node are normal, which indicates that the network of the first node and the second node is normal. The detection result of the first local IO stream detection is abnormal, which indicates that the local disk of the first node may have a problem of being unable to read and write normally (it is also possible that the IO stream detection service of the first node is abnormal). However, the detection result of the second peer IO stream detection is normal, which indicates that the local system disk of the first node can be normally read and written. At this time, the case that the RAID card state of the first node is abnormal can be excluded, and it can be determined that the abnormal detection result of the first local IO stream detection is that the IO stream detection service of the first node is abnormal. The detection result of the second local IO stream detection is normal, which indicates that the local disk of the second node can be normally read and written. However, the detection result of the first peer IO stream detection is abnormal, which further verifies that the IO stream detection of the first node is abnormal.

[0095] Similarly, when the heartbeat detection results of the first node and the second node are both normal, the detection result of the first local IO stream detection is normal, the detection result of the first peer IO stream detection is normal, the detection result of the second local IO stream detection is abnormal, and the detection result of the second peer IO stream detection is abnormal, it is determined that the IO stream detection service of the second node is abnormal.

[0096] In some implementations, the network status of the first node and the second node also affects the detection result of the peer IO stream detection. When the heartbeat detection result of the first node and / or the second node is abnormal, the detection result of the peer IO stream detection is also abnormal.

[0097] Step S502: When it is determined that the RAID card state of the master node is abnormal and the RAID card state of the standby node is normal, the cluster service is switched from the master node to the standby node for running, and alarm processing of the RAID card soft failure of the master node is performed.

[0098] Step S503: When the cluster service system is in the hot standby scenario, the RAID card state of the standby node is determined to be abnormal, and the RAID card state of the master node is normal, the alarm processing of the standby node RAID card soft failure is performed.

[0099] When the cluster service system is in the dual-active scenario, the RAID card state of the standby node is determined to be abnormal, and the RAID card state of the master node is normal, the cluster service is switched from the standby node to the master node for running, and the alarm processing of the standby node RAID card soft failure is performed.

[0100] Step S504: When the RAID card state of the master node is determined to be abnormal, and the RAID card state of the standby node is abnormal, the alarm processing of the master node and the standby node RAID card soft failure is performed.

[0101] Step S505: When the IO flow detection of the master node and / or the standby node is determined to be abnormal, the alarm processing of the IO flow detection failure of the master node and / or the standby node is performed.

[0102] Specifically, the IO flow detection abnormality indicates that the IO test tool, such as FIO, IOmeter, etc., has failed, and the IO flow detection failure needs to be alarmed to enable the management personnel to handle the IO flow detection failure.

[0103] In an example, the cluster service system is in a hot standby scenario, including a first node and a second node, the first node is a master node, the second node is a standby node, and a database service runs on the first node. When the first node fails, the database service is switched from the first node to the second node for running. The cluster service system can also include a management node. The first node or the management node determines the RAID card state of the first node and the second node according to the detection results of the first local IO flow detection, the first peer IO flow detection, the second local IO flow detection, and the second peer IO flow detection, and the heartbeat detection results of the first node and the second node, and processes the cluster service according to the RAID card states of the first node and the second node. The first node is a master node, and the second node is a standby node. The specific mode is shown in Table 1 as follows:

[0104] Table 1: IO flow detection results and processing scheme of the first node and the second node in the hot standby scenario

[0105]

[0106]

[0107] In one example, the cluster service system operates in an active-active scenario, comprising a first node and a second node, which serve as backup nodes for each other. The first node runs the first cluster service, and the second node runs the second cluster service. When the first node fails, the first cluster service switches to the second node; conversely, when the second node fails, the second cluster service switches to the first node. This cluster service system may also include a management node.

[0108] The first node, second node, or management node acquires the detection results of the first local IO stream detection and the first peer IO stream detection, the detection results of the second local IO stream detection and the second peer IO stream detection, as well as the heartbeat detection results of the first node and the second node. Based on the detection results of the first local IO stream detection, the first peer IO stream detection, the second local IO stream detection, the second peer IO stream detection, and the heartbeat detection results of the first node and the second node, the first node, the second node, or management node determines the RAID card status of the first node and the second node respectively, and processes the cluster services according to the RAID card status of the first node and the second node. The specific method is shown in Table 2 below:

[0109] Table 2. IO flow detection results and processing solutions for the first and second nodes in a dual-active scenario.

[0110]

[0111]

[0112] This explanation will focus on a dual-active scenario where the RAID status of the first node is abnormal, while the RAID status of the second node is normal. Figure 6 This is a flowchart illustrating a fourth embodiment of a cluster service processing method provided in this application. Figure 6 As shown, the first node and the second node serve as backup nodes for each other. Database service A runs on the first node, and database service B runs on the second node. The heartbeat detection results of both the first and second nodes are normal. However, the detection result of the first local IO stream is abnormal, while the detection result of the first peer IO stream is normal. The detection result of the second local IO stream is normal, but the detection result of the second peer IO stream is abnormal. Therefore, it is determined that the system disk read of the first node is abnormal, indicating a system freeze. Database service A needs to be switched from the first node to the second node.

[0113] Further, since the operation of the operating system can be affected by the soft failure of the RAID card, and the cluster service cannot be switched normally, it is necessary to determine whether the cluster service is switched successfully. If the switching is successful, the switching process is ended; if the switching fails, the first node can restart the second node through an out-of-band management channel, so that the cluster service is switched from the second node to the first node. Exemplarily, the out-of-band management channel can be an Intelligent Platform Management Interface (IPMI) channel. Figure 7 A flowchart of an embodiment of a cluster service processing method provided by the present application.

[0114] As shown in Figure 7 , the present application periodically performs read operations on the system disk by calling an underlying IO test tool, such as FIO, IOmeter, etc., to perform IO flow detection. Exemplarily, if the returned result is normal; if it is not responded after timeout, it is determined that the system disk read is abnormal, and there is a system hang situation, and the result needs to be synchronized to the node where the cluster service is located to realize the switching of the cluster service to the normal node to ensure the service continuity. If the node where the cluster service is located at this time cannot realize the switching under the hung system, a network channel, such as an IPMI channel, can be established between the normal node and the fault node through the out-of-band management to realize the restart of the fault node and thus realize the switching of the cluster service.

[0115] In the present embodiment, the first node determines the RAID card states of the first node and the second node according to the detection results of the first local IO flow detection, the first peer IO flow detection, the second local IO flow detection and the second peer IO flow detection, and the heartbeat detection results of the first node and the second node, and processes the cluster service according to the RAID card states of the first node and the second node, which can realize the alarm of abnormal situations on the basis of the switching of the service from the fault node to the normal node in time when the service needs to be switched due to the soft failure of the RAID card. Further, the problem that the service cannot be switched from the node where the RAID card fails to the normal node due to the fact that the heartbeat network detection cannot detect the soft failure of the RAID card in the node in the prior art is solved.

[0116] The following is an apparatus embodiment of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the apparatus embodiments of the present application, please refer to the method embodiments of the present application.

[0117] Figure 8 A structure diagram of an embodiment of a server provided by the present application; as Figure 8As shown, the server 60 comprises an acquisition module 61 and a processing module 62. The acquisition module 61 is configured to acquire a detection result of local input / output (IO) flow detection when the heartbeat network of the first node and the heartbeat network of the second node are normal, and the acquisition module 61 is further configured to acquire a detection result of opposite-end IO flow detection when the detection result of the local IO flow detection is abnormal. The processing module 62 is configured to determine a fault cause based on the detection result of the opposite-end IO flow detection and the detection result of the local IO flow detection, and perform processing on cluster services according to the fault cause. The local IO flow detection is first local IO flow detection of IO flow between a first redundant array of independent disks (RAID) card and a first disk in the first node by the first node, and / or second local IO flow detection of IO flow between a second RAID card and a second disk in the second node by the second node. The opposite-end IO flow detection is first opposite-end IO flow detection of IO flow between the second RAID card and the second disk in the second node by the first node, and / or second opposite-end IO flow detection of IO flow between the first RAID card and the first disk in the first node by the second node.

[0118] The server provided by the embodiments of the present application can execute the technical solutions shown in the method embodiments, and the implementation principles and beneficial effects are similar, which will not be repeated here.

[0119] In a possible implementation, the acquisition module 61 is specifically configured to initiate a first read data instruction to a disk of a local node by the first node and the second node respectively, wherein the first read data instruction is used to read first data in the disk of the local node, and the acquisition module 61 is further configured to acquire first data returned by the disk of the local node based on the first read data instruction. The detection result of the local IO flow detection is abnormal, including that the first data returned by the disk of the local node is not acquired.

[0120] In a possible implementation, the acquisition module 61 is specifically configured to acquire the first data returned by the disk of the local node based on the first read data instruction at preset time interval points within a preset first time period. The detection result of the local IO flow detection is abnormal, and further includes that if the sum of the number of times of acquiring the first data returned by the disk of the local node at the preset time interval points within the preset first time period is less than a first number threshold, it is determined that the detection result of the local IO flow detection is abnormal.

[0121] In a possible implementation, the obtaining module 61 is specifically configured to obtain the first data returned by the disk of the local node based on the first read data instruction within a preset second time period; and the detection result of the local IO stream detection is abnormal, and the method further includes: if a difference between an obtaining time of the first data returned by the disk of the local node and the preset second time is greater than a first time threshold, it is determined that the detection result of the local IO stream detection is abnormal; and the preset second time is a maximum time of normally obtaining the first data returned by the disk of the local node.

[0122] In a possible implementation, the obtaining module 61 is specifically configured to obtain the first data returned by the disk of the local node based on the first read data instruction within a preset first time period; and the detection result of the local IO stream detection is abnormal, and the method further includes: if a difference between an obtaining time of the first data returned by the disk of the local node and a preset second time is greater than a first time threshold for more than a second time threshold, it is determined that the detection result of the local IO stream detection is abnormal; and the preset second time is a maximum time of normally obtaining the first data returned by the disk of the local node.

[0123] The server provided by the embodiments of the present application can execute the technical solutions shown in the method embodiments, and the implementation principles and beneficial effects are similar, and will not be repeated here.

[0124] In a possible implementation, the obtaining module 61 is specifically configured to initiate, by the first node or the second node, a second read data instruction to the disk in the opposite node; the second read data instruction is used to read second data in the disk in the opposite node; obtain the second data returned by the disk in the opposite node based on the second read data instruction; if the second data returned by the disk in the opposite node is obtained, it is determined that the detection result of the opposite IO stream detection is normal; and if the second data returned by the disk in the opposite node is not obtained, it is determined that the detection result of the opposite IO stream detection is abnormal.

[0125] The server provided by the embodiments of the present application can execute the technical solutions shown in the method embodiments, and the implementation principles and beneficial effects are similar, and will not be repeated here.

[0126] In a possible implementation, the processing module 62 is specifically configured to: in a case where the detection result of the first local IO flow detection is abnormal and the detection result of the first peer IO flow detection is normal, determine that the RAID state of the first node is abnormal; in a case where the detection result of the second local IO flow detection is abnormal and the detection result of the second peer IO flow detection is normal, determine that the RAID state of the second node is abnormal; in a case where the detection result of the first local IO flow detection is abnormal, the detection result of the first peer IO flow detection is abnormal, and the detection result of the second local IO flow detection is normal, determine that the IO flow detection service of the first node is abnormal; and in a case where the detection result of the second local IO flow detection is abnormal, the detection result of the second peer IO flow detection is abnormal, and the detection result of the first local IO flow detection is normal, determine that the IO flow detection service of the second node is abnormal.

[0127] The server provided by the embodiments of the present application can execute the technical solutions shown in the method embodiments, and the implementation principles and beneficial effects are similar, which will not be repeated here.

[0128] In a possible implementation, the cluster service system is in a hot standby scene, the first node is a master node, and the second node is a standby node. The processing module 62 is specifically configured to: in a case where it is determined that the RAID card state of the first node is abnormal and the RAID card state of the second node is normal, switch the cluster service from the first node to the second node for running, and perform alarm processing of a soft failure of the RAID card of the first node; in a case where it is determined that the RAID card state of the second node is abnormal and the RAID card state of the first node is normal, perform alarm processing of a soft failure of the RAID card of the second node; in a case where it is determined that the RAID card state of the first node is abnormal and the RAID card state of the second node is abnormal, perform alarm processing of soft failures of the RAID cards of the first node and the second node; and in a case where it is determined that the IO flow detection service of the first node and / or the second node is abnormal, perform alarm processing of an IO flow detection failure of the first node and / or the second node.

[0129] In one possible implementation, the cluster service system is in a dual-active scenario, with the first node and the second node serving as backup nodes for each other. The first node runs the first cluster service, and the second node runs the second cluster service. Specifically, the processing module 62 is configured to: when it is determined that the RAID card status of the first node is abnormal, and the RAID card status of the second node is normal, switch the first cluster service from the first node to the second node and perform alarm processing for a soft fault of the RAID card of the first node; when it is determined that the RAID card status of the second node is abnormal, and the RAID card status of the first node is normal, switch the second cluster service from the second node to the first node and perform alarm processing for a soft fault of the RAID card of the second node; when it is determined that the RAID card status of both the first and second nodes is abnormal, perform alarm processing for soft faults of the RAID cards of both nodes; and when it is determined that the IO stream detection service of the first node and / or the second node is abnormal, perform alarm processing for IO stream detection failure of the first node and / or the second node.

[0130] The server provided in this application embodiment can execute the technical solutions shown in the above method embodiments. Its implementation principle and beneficial effects are similar, and will not be repeated here.

[0131] Figure 9 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Figure 9 As shown, the server 70 includes a processor 71, a memory 72, and a communication interface 73; wherein the memory 72 is used to store executable instructions of the processor 71; the processor 71 is configured to execute the technical solutions in any of the foregoing method embodiments by executing the executable instructions.

[0132] Optionally, the memory 72 can be either standalone or integrated with the processor 71.

[0133] Optionally, when the memory 72 is a device independent of the processor 71, the server 70 may further include a bus 74 for connecting the aforementioned devices.

[0134] The server is used to execute the technical solutions in any of the aforementioned method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0135] This application also provides a cluster service system. The cluster service system includes at least one first node and at least one second node, wherein the first node is the master node and the second node is the backup node; wherein the first node executes the technical solutions in any of the foregoing method embodiments.

[0136] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. The foregoing program can be stored in a computer readable storage medium. The program executes to perform the steps of the above-mentioned method embodiments; and the foregoing storage medium includes various storage media that can store program codes, such as ROM, RAM, magnetic disk or optical disk.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of cluster service processing, characterized by, The method is applied to a cluster service system, and the cluster service system comprises a first node and a second node. In a case where heartbeat networks of the first node and the second node are normal, a detection result of local input / output (IO) flow detection is obtained; the local IO flow detection is achieved by issuing a first read data instruction to a disk of a local node. In a case where the detection result of the local IO flow detection is abnormal, a detection result of peer IO flow detection is obtained. Based on the detection result of the peer IO flow detection and the detection result of the local IO flow detection, a fault cause is determined, and the cluster service is processed according to the fault cause. The local IO flow detection is first local IO flow detection of IO flow between a first independent redundant array of inexpensive disks (RAID) card and a first disk in the first node by the first node, and second local IO flow detection of IO flow between a second RAID card and a second disk in the second node by the second node. The peer IO flow detection is first peer IO flow detection of IO flow between the second RAID card and the second disk in the second node by the first node, and / or second peer IO flow detection of IO flow between the first RAID card and the first disk in the first node by the second node.

2. The cluster service processing method of claim 1, wherein, The detection result of the local IO flow detection is obtained by comprising: The first node and the second node respectively issue a first read data instruction to a disk of a local node; the first read data instruction is used to read first data in the disk in the local node. First data returned by the disk of the local node based on the first read data instruction is obtained. The detection result of the local IO flow detection is abnormal, comprising: no first data returned by the disk of the local node is obtained.

3. The cluster service processing method of claim 2, wherein, The first data returned by the disk of the local node based on the first read data instruction is obtained by comprising: Within a preset first time period, the first data returned by the disk of the local node based on the first read data instruction is obtained at preset time interval points; The detection result of the local IO flow detection is abnormal, further comprising: if a total number of times of obtaining the first data returned by the disk of the local node within the preset first time period at the preset time interval points is less than a first number threshold, it is determined that the detection result of the local IO flow detection is abnormal.

4. The cluster service processing method of claim 2, wherein, The first data returned by the disk of the local node based on the first read data instruction is obtained by comprising: Within a preset second time period, the first data returned by the disk of the local node based on the first read data instruction is obtained. The detection result of the local IO flow detection is abnormal, further comprising: if a difference between an obtaining time of the first data returned by the disk of the local node and the preset second time is greater than a first time threshold, it is determined that the detection result of the local IO flow detection is abnormal; The preset second time is a maximum time of normally obtaining the first data returned by the disk of the local node.

5. The cluster service processing method of claim 2, wherein, The first data returned by the disk of the local node based on the first read data instruction comprises: In a preset first time period, the first data returned by the disk of the local node based on the first read data instruction is acquired; The detection result of the local IO flow detection is abnormal, and further comprising: in the preset first time period, the difference between the acquisition time of the first data returned by the disk of the local node and a preset second time is greater than a first time threshold value for a number of times greater than a second number threshold value; the preset second time is the maximum time for normally acquiring the first data returned by the disk of the local node.

6. The cluster service processing method of claim 1, wherein, The detection result of the opposite end IO flow detection comprises: The first node or the second node initiates a second read data instruction to the disk of the opposite end node; wherein the second read data instruction is used to read second data in the disk of the opposite end node; The second data returned by the disk of the opposite end node based on the second read data instruction is acquired; if the second data returned by the disk of the opposite end node is acquired, it is determined that the detection result of the opposite end IO flow detection is normal; if the second data returned by the disk of the opposite end node is not acquired, it is determined that the detection result of the opposite end IO flow detection is abnormal.

7. The cluster service processing method of claim 1, wherein: The detection result of the opposite end IO flow detection comprises: In the case that the first local IO flow detection result is abnormal and the first opposite end IO flow detection result is normal, it is determined that the RAID state of the first node is abnormal; In the case that the second local IO flow detection result is abnormal and the second opposite end IO flow detection result is normal, it is determined that the RAID state of the second node is abnormal; In the case that the first local IO flow detection result is abnormal, the first opposite end IO flow detection result is abnormal, and the second local IO flow detection result is normal, it is determined that the IO flow detection service of the first node is abnormal; In the case that the second local IO flow detection result is abnormal, the second opposite end IO flow detection result is abnormal, and the first local IO flow detection result is normal, it is determined that the IO flow detection service of the second node is abnormal.

8. The cluster service processing method of claim 7, wherein, The cluster service is processed according to the fault reason, comprising: The cluster service system is in a hot standby scenario, the first node is a master node, and the second node is a standby node; When it is determined that the RAID card state of the first node is abnormal and the RAID card state of the second node is normal, the cluster service is switched from the first node to the second node for running, and alarm processing of the first node RAID card soft fault is performed; When it is determined that the RAID card state of the second node is abnormal and the RAID card state of the first node is normal, alarm processing of the second node RAID card soft fault is performed; when it is determined that the RAID card of the first node is abnormal and the RAID card of the second node is abnormal, performing alarm processing of RAID card soft failure of the first node and the second node; when it is determined that the IO flow detection service of the first node and / or the second node is abnormal, performing alarm processing of IO flow detection failure of the first node and / or the second node.

9. The cluster service processing method of claim 7, wherein, the processing of the cluster service according to the failure cause comprises: the cluster service system is in a dual-active scenario, the first node and the second node are backup nodes of each other, the first node runs a first cluster service, and the second node runs a second cluster service; when it is determined that the RAID card of the first node is abnormal and the RAID card of the second node is normal, switching the first cluster service from the first node to the second node to run, and performing alarm processing of RAID card soft failure of the first node; when it is determined that the RAID card of the second node is abnormal and the RAID card of the first node is normal, switching the second cluster service from the second node to the first node to run, and performing alarm processing of RAID card soft failure of the second node; when it is determined that the RAID card of the first node is abnormal and the RAID card of the second node is abnormal, performing alarm processing of RAID card soft failure of the first node and the second node; when it is determined that the IO flow detection service of the first node and / or the second node is abnormal, performing alarm processing of IO flow detection failure of the first node and / or the second node.

10. A cluster services system, characterized by comprise: at least one first node and at least one second node, wherein the first node is a master node, and the second node is a backup node; wherein the first node performs the cluster service processing method of any one of claims 1 to 9.

Images