Distributed storage file system fault tolerance method, product, equipment and storage medium

Abstract

The invention relates to a distributed storage file system fault tolerance method, a product, equipment and a storage medium. The method comprises the steps that all nodes in a shared cluster connected to a heartbeat disk are obtained, read-write statistical data of each node at each magnetic disk heartbeat are stored as slot data on the heartbeat disk, and the read-write statistical data comprise normal or abnormal read-write input and output and the number of times of continuous occurrence of read-write input and output abnormities; when the heartbeat disk heartbeat each time, responding to the read-write input and output data received by the first node, updating the slot data of the first node, and reading and verifying the slot data of the second node in the shared cluster; and judging whether the number of times of continuously occurring read-write input and output abnormities in the slot data of the second node is greater than a first threshold value or not, if so, controlling the second node to be offline, and if not, checking the slot data of the next node. The problem that the cluster is unavailable due to accidental IO errors is avoided.

Description

Technical Field

[0001] The present application relates to the technical field of computer data storage, and in particular to a distributed storage file system fault tolerance method, product, computer device and storage medium. Background Art

[0002] Cluster systems are usually designed to share network and disk resources and use node heartbeats to communicate with each other to maintain services within the cluster. Cluster systems usually include the ability to detect non-responsive nodes and remove them from the cluster to maintain their ability to continue to provide services and avoid failures or data corruption. OCFS2 is a distributed storage file system that utilizes shared storage, and it can be used for many general computing tasks that require shared cluster storage. OCFS2 uses disk-based heartbeats to determine whether nodes within the cluster are available. If a node is unresponsive, other nodes in the cluster can continue to provide available services by removing the node from the cluster. If a node cannot communicate through shared storage, the node will be isolated, which means that it will be expelled from the cluster so that the cluster can return to a running state. The reason for the abnormal function of the OCFS2 cluster is that the existing heartbeat mechanism has too strict requirements on IO reading and writing, and cannot tolerate occasional IO errors, resulting in nodes in the OCFS2 cluster being expelled from the cluster due to occasional IO errors, causing frequent node reconstruction or causing the entire OCFS2 cluster to be unavailable. Summary of the invention

[0003] Based on this, a distributed storage file system fault tolerance method, apparatus, computer equipment and storage medium are provided, which can prevent nodes in a distributed storage file system cluster from being expelled from the cluster due to occasional IO errors, increase the IO fault tolerance mechanism, and avoid the problem of occasional IO errors causing the entire distributed storage file system cluster to be unavailable.

[0004] On the one hand, a distributed storage file system fault tolerance method is provided, the method comprising: Obtain all nodes in the shared cluster connected to the heartbeat disk, and store the read and write statistics of each node in each disk heartbeat as slot data on the heartbeat disk, wherein the read and write statistics include whether the read and write input and output are normal or abnormal, and the number of consecutive read and write input and output abnormalities; At each disk heartbeat of the heartbeat disk, in response to the first node receiving read / write input / output data, updating the slot data of the first node, reading the slot data of the second node in the shared cluster, filtering and saving the valid generation information of the second node according to the root field in the slot data of the second node, and counting the number of consecutive read / write input / output anomalies; Determine whether the number of consecutive read / write input / output anomalies in the slot data of the second node is greater than a first threshold. If so, control the second node to go offline. Otherwise, check the slot data of the next node.

[0005] In one embodiment, storing the read and write statistics of each node in each disk heartbeat via slot data on the heartbeat disk includes: Setting the initial value of the number of consecutive read / write input / output exceptions to zero; Determine whether the read and write input and output of the current node is normal or abnormal during the current disk heartbeat; In response to the current node reading and writing input and output being normal during the current disk heartbeat, determining whether the current node reading and writing input and output is normal or abnormal during the next heartbeat reading and writing; In response to a read / write input / output exception of the current node during a current disk heartbeat, the number of consecutive read / write input / output exceptions of the current node is increased by one.

[0006] In one embodiment, determining whether the read / write input / output of the current node is normal or abnormal during the current disk heartbeat includes: The generation information of each node is stored on the heartbeat disk, and the read and write statistics also include the generation information of the current node performing the reading and writing; Read the heartbeat read and write statistics of the second node that shares a connection with the first node, and determine whether the generation information of the current node performing reading and writing in the read and write statistics is the same as the generation information of the second node stored on the heartbeat disk; if they are different, determine that the current read and write input and output are abnormal; if they are the same, determine that the current read and write input and output are normal.

[0007] In one embodiment, determining whether the read / write input / output of the current node is normal or abnormal during the current disk heartbeat further includes: Determine whether there is an error in the data read and written in the read and write statistical data; If there is no error and the generation information of the current node being read and written in the read and write statistical data is the same as the generation information of the second node stored on the heartbeat disk, it is determined that the current read and write input and output are normal; If there is an error and the generation information of the current node being read and written in the read and write statistical data is the same as the generation information of the second node stored on the heartbeat disk, the current read and write input and output error is ignored.

[0008] In one embodiment, the reading of the slot data of the second node in the shared cluster, filtering and saving the valid generation information of the second node according to the root field in the slot data of the second node, and counting the number of consecutive read / write input / output exceptions comprises: The slot data is set to further include a first root field, a second root field, a third root field and a fourth root field, the initial value of the slot data is zero, the first root field is used to record the valid generation information of the corresponding node, the second root field is used to record the number of consecutive identical generation information read from the corresponding node, the third root field is used to record the number of consecutive different generation information of each node, and the fourth root field is used to record whether the generation information of each node has been authenticated as legal; In response to reading the slot data of the second node, obtaining the value of the fourth root field of the second node, if the value of the fourth root field of the second node is zero, determining that the first node does not store the valid generation information of the second node, and if the value of the fourth root field of the second node is not zero, determining that the first node has stored the valid generation information of the second node; In response to the first node not storing the valid generation information of the second node, obtaining the value of the second root field of the second node, and writing the node generation information corresponding to the normal read and write input and output into the first root field as the valid generation information of the second node by judging whether the second root field of the second node is greater than a third threshold; In response to the first node having saved the generation information of the second node, the value of the third root field of the second node is obtained, and the number of consecutive read and write input and output exceptions in the slot data of the second node is set by determining whether the third root field of the second node is greater than a second threshold.

[0009] In one embodiment, the fourth root field is used to record whether the generation information of each node has been authenticated as legitimate and includes: Setting the fourth root field to zero or one; In response to obtaining that the fourth root field is one, determining that the generation information of the corresponding node has been authenticated as legal; In response to obtaining that the fourth root field is zero, it is determined that the generation information of the corresponding node is not authenticated as legal.

[0010] In one embodiment, the step of judging whether the second root field of the second node is greater than a third threshold value to write the node generation information corresponding to normal read / write input and output into the first root field as the valid generation information of the second node comprises: Determine whether the value of the second root field of the second node is greater than a third threshold, if so, set the fourth root field to one, if not, determine whether the second root field of the second node is zero; If the second root field of the second node is zero, the generation information of the second node is read from the heartbeat disk and stored in the first root field, and the second root field is incremented by one; If the second root field of the second node is not zero, indicating that the generation information of the second node has been stored in the first root field, then determine whether the value of the first root field of the second node in the slot data is the same as the generation information of the second node read from the heartbeat disk, if they are the same, add one to the second root field, if they are different, save the generation information of the second node read from the heartbeat disk to the first root field, and set the second root field to one; If the value of the second root field of the second node plus one is greater than the third threshold, the fourth root field is set to one.

[0011] In one embodiment, when the value of the second root field of the second node is less than or equal to a third threshold, before determining whether the second root field of the second node is zero, the method further includes: Determine whether the generation information of the second node read from the heartbeat disk is zero; If it is zero, end; if it is not zero, determine whether the second root field of the second node is zero.

[0012] In one embodiment, the step of setting the number of consecutive read / write input / output exceptions in the slot data of the second node by determining whether the third root field of the second node is greater than a second threshold value comprises: If the value of the fourth root field of the second node is one, determine whether the value of the first root field of the second node in the slot data is the same as the generation information of the second node read from the heartbeat disk, and simultaneously determine whether the value of the third root field of the second node is zero; If the value of the first root field of the second node in the slot data is the same as the generation information of the second node read from the heartbeat disk, and the value of the third root field of the second node is not zero and is less than the second threshold, then the third root field is set to zero; If the value of the first root field of the second node in the slot data is different from the generation information of the second node read from the heartbeat disk, the third root field count is increased by one, and it is determined whether the third root field after increase is greater than the second threshold. If so, the number of consecutive read / write input / output exceptions in the slot data of the second node is increased by one, otherwise the process ends.

[0013] In one of the embodiments, when controlling the second node to go offline, the method further includes: The slot data of the second node is cleared to trigger a command to kick out or isolate the second node from the shared cluster.

[0014] In one of the embodiments, the distributed storage file system fault tolerance method further includes: Before the heartbeat disk performs the next disk heartbeat, it is determined whether the slot data of all nodes have been verified, and if so, the process ends; otherwise, the slot data of the next node is verified.

[0015] In one of the embodiments, the distributed storage file system fault tolerance method further includes: In response to the first node completing the slot data verification of all nodes, the second node is controlled to receive read and write input and output data, the slot data of the second node is updated, and the slot data of the third node in the shared cluster is read and verified until all nodes in the shared cluster connected to the heartbeat disk complete receiving the read and write input and output data.

[0016] In another aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps: Obtain all nodes in the shared cluster connected to the heartbeat disk, and store the read and write statistics of each node in each disk heartbeat as slot data on the heartbeat disk, wherein the read and write statistics include whether the read and write input and output are normal or abnormal, and the number of consecutive read and write input and output abnormalities; At each disk heartbeat of the heartbeat disk, in response to the first node receiving read / write input / output data, updating the slot data of the first node, reading the slot data of the second node in the shared cluster, filtering and saving the valid generation information of the second node according to the root field in the slot data of the second node, and counting the number of consecutive read / write input / output anomalies; Determine whether the number of consecutive read / write input / output anomalies in the slot data of the second node is greater than a first threshold. If so, control the second node to go offline. Otherwise, check the slot data of the next node.

[0017] In another aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented: Obtain all nodes in the shared cluster connected to the heartbeat disk, and store the read and write statistics of each node in each disk heartbeat as slot data on the heartbeat disk, wherein the read and write statistics include whether the read and write input and output are normal or abnormal, and the number of consecutive read and write input and output abnormalities; At each disk heartbeat of the heartbeat disk, in response to the first node receiving read / write input / output data, updating the slot data of the first node, reading the slot data of the second node in the shared cluster, filtering and saving the valid generation information of the second node according to the root field in the slot data of the second node, and counting the number of consecutive read / write input / output anomalies; Determine whether the number of consecutive read / write input / output anomalies in the slot data of the second node is greater than a first threshold. If so, control the second node to go offline. Otherwise, check the slot data of the next node.

[0018] In another aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: Obtain all nodes in the shared cluster connected to the heartbeat disk, and store the read and write statistics of each node in each disk heartbeat as slot data on the heartbeat disk, wherein the read and write statistics include whether the read and write input and output are normal or abnormal, and the number of consecutive read and write input and output abnormalities; At each disk heartbeat of the heartbeat disk, in response to the first node receiving read / write input / output data, updating the slot data of the first node, reading the slot data of the second node in the shared cluster, filtering and saving the valid generation information of the second node according to the root field in the slot data of the second node, and counting the number of consecutive read / write input / output anomalies; Determine whether the number of consecutive read / write input / output anomalies in the slot data of the second node is greater than a first threshold. If so, control the second node to go offline. Otherwise, check the slot data of the next node.

[0019] The above-mentioned distributed storage file system fault-tolerant method, product, computer equipment and storage medium use all nodes in the shared cluster connected to the heartbeat disk to interactively check whether the read and write input and output data are abnormal, and set a setting that the corresponding node is controlled offline only when the number of consecutive read and write input and output anomalies is greater than a first threshold, thereby increasing the input and output fault-tolerant mechanism and avoiding the problem of occasional input and output errors causing the entire distributed storage file system cluster to be unavailable. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 It is a schematic diagram of the heartbeat mechanism of the OCFS2 file system in the prior art; Figure 2 This is a flow chart of a distributed storage file system fault tolerance method in one embodiment of the present application; Figure 3 A logical diagram of a distributed storage file system fault tolerance method in one embodiment of the present application; Figure 4A schematic diagram of a structure in which M groups of gen fields are added to a memory for temporarily storing status information of gen fields in node slot data in one embodiment of the present application; Figure 5 A flowchart of a step of judging whether the second root field of the second node is greater than a third threshold value so as to write the corresponding node generation information when the read and write input and output are normal into the first root field as the valid generation information of the second node in one embodiment of the present application; Figure 6 A flowchart of the step of setting the number of consecutive read / write input / output exceptions in the slot data of the second node by judging whether the third root field of the second node is greater than a second threshold value in one embodiment of the present application; Figure 7 This is a structural block diagram of a distributed storage file system fault tolerance device in one embodiment of the present application; Figure 8 This is a diagram of the internal structure of a computer device in one embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0023] As described in the background technology, OCFS2 is a distributed storage file system that uses shared storage and can be used for many general computing tasks that require shared cluster storage. In OCFS2, each node has its own sector slot on the shared heartbeat disk. Under normal circumstances, when a node joins the shared heartbeat disk, it writes a unique identifier "generation information" to its own slot, and then periodically writes timestamps and other information to its own slot, and periodically reads the slot data of other nodes to obtain the current status of other nodes.

[0024] like Figure 1As shown in the figure, taking a two-node cluster as an example, the heartbeat processing logic and the impact of occasional heartbeat anomalies. Node A and Node B share a heartbeat disk for disk heartbeat. First, node A updates the data in its own slot A, and then reads the data in slot B belonging to node B. If all 0 data is read at this time due to a shared disk failure or high disk pressure, node B will be considered unavailable and isolated from the cluster. Similarly, node B updates the data in its own slot B, and then reads the data in slot A belonging to node A. The data verification is correct, and node A is considered available. Then a message is sent to node A. At this time, node A thinks that node B is no longer in the cluster, so it will not receive messages from node B, so that the message sent by node B to node A is blocked, which in turn affects the OCFS2 cluster function. The reason for the abnormal function of the OCFS2 cluster is that the existing heartbeat mechanism has too strict requirements on IO reading and writing, and cannot tolerate occasional IO errors.

[0025] Therefore, under the current OCFS2 disk heartbeat mechanism, how to increase the IO fault tolerance mechanism to avoid occasional IO errors causing the entire OCFS2 cluster to be unavailable is a technical problem that needs to be solved.

[0026] To solve the above problems, an embodiment of the present invention creatively proposes a distributed storage file system fault tolerance method. The heartbeat situation classification and expected effects are shown in Table 1.

[0027] Table 1 Heart rate classification table

[0028] In the heartbeat count column, one heartbeat includes one write IO and one read IO.

[0029] The caseN column indicates different heartbeat read IO exceptions, where T indicates that the read IO is correct, F indicates that the read IO is abnormal, and - indicates ignore.

[0030] The result line indicates the impact in the current case. ok means the cluster function is not affected, old means the cluster function is affected in the traditional solution, tolerate means the cluster function is not affected in the current solution, and fail means the cluster function is affected in the current solution.

[0031] The detailed instructions are as follows: Case 1: If the disk heartbeat read IO is normal every time, the cluster function is normal; Case 2: In the traditional solution, if a heartbeat read IO exception occurs occasionally, the cluster function will be affected; Case 3-8: Under the current solution, if the number of consecutive heartbeat read IO exceptions does not exceed 2, it can be tolerated and the cluster functions normally; Case 9: Under the current solution, if the continuous heartbeat read IO exception occurs more than 2 times, the cluster function will be affected.

[0032] In one embodiment, Figure 2 , Figure 3 As shown, a distributed storage file system fault tolerance method is provided, comprising the following steps: Step S1, obtaining all nodes in the shared cluster connected to the heartbeat disk, and storing the read and write statistics of each node in each disk heartbeat as slot data on the heartbeat disk, wherein the read and write statistics include whether the read and write input and output are normal or abnormal, and the number of consecutive read and write input and output abnormalities; Step S2, in response to the first node receiving the read / write input / output data at each disk heartbeat of the heartbeat disk, updating the slot data of the first node, reading the slot data of the second node in the shared cluster, filtering and saving the valid generation information of the second node according to the root field in the slot data of the second node, and counting the number of consecutive read / write input / output anomalies; Step S3, determining whether the number of consecutive read / write input / output anomalies in the slot data of the second node is greater than a first threshold value, if so, controlling the second node to go offline, otherwise checking the slot data of the next node.

[0033] Among them, all nodes in the shared cluster connected to the heartbeat disk are used to interactively check whether the read and write input and output data are abnormal, and the corresponding node is controlled to be offline only when the number of consecutive read and write input and output anomalies is greater than the first threshold, thereby increasing the input and output fault tolerance mechanism and avoiding the problem of occasional input and output errors causing the entire distributed storage file system cluster to be unavailable.

[0034] In this embodiment, the storage of the read and write statistics of each node in each disk heartbeat on the heartbeat disk through slot data includes: Setting the initial value of the number of consecutive read / write input / output exceptions to zero; Determine whether the read and write input and output of the current node is normal or abnormal during the current disk heartbeat; In response to the current node reading and writing input and output being normal during the current disk heartbeat, determining whether the current node reading and writing input and output is normal or abnormal during the next heartbeat reading and writing; In response to a read / write input / output exception of the current node during a current disk heartbeat, the number of consecutive read / write input / output exceptions of the current node is increased by one.

[0035] Among them, by continuously accumulating the number of abnormal read and write input and output times, the first threshold is set to achieve IO abnormal fault tolerance. The first threshold represents the number of continuous IO errors that can be tolerated. It is configured when the distributed storage file system (OCFS2) is started. The value is required to be greater than 0. The larger the value, the stronger the IO fault tolerance capability. It is recommended to configure the first threshold to 3. Implement Case 3-8: Under the current solution, if the continuous heartbeat read IO abnormality does not exceed 2 times, it can be tolerated and the cluster function is normal; Case 9: Under the current solution, if the continuous heartbeat read IO abnormality exceeds 2 times, the cluster function is affected.

[0036] In this embodiment, determining whether the read / write input / output of the current node is normal or abnormal during the current disk heartbeat includes: The generation information of each node is stored on the heartbeat disk, and the read and write statistics also include the generation information of the current node performing the reading and writing; Read the heartbeat read and write statistics of the second node that shares a connection with the first node, and determine whether the generation information of the current node performing reading and writing in the read and write statistics is the same as the generation information of the second node stored on the heartbeat disk; if they are different, determine that the current read and write input and output are abnormal; if they are the same, determine that the current read and write input and output are normal.

[0037] Among them, whether the generation information of the node during the reading and writing process is the same as the generation information of the node stored on the heartbeat disk is used to judge whether the current reading and writing input and output is normal or abnormal, thereby avoiding interference from abnormal input and output data.

[0038] In this embodiment, determining whether the read / write input / output of the current node is normal or abnormal during the current disk heartbeat also includes: Determine whether there is an error in the data read and written in the read and write statistical data; If there is no error and the generation information of the current node being read and written in the read and write statistical data is the same as the generation information of the second node stored on the heartbeat disk, it is determined that the current read and write input and output are normal; If there is an error and the generation information of the current node being read and written in the read and write statistical data is the same as the generation information of the second node stored on the heartbeat disk, the current read and write input and output error is ignored.

[0039] Combined with Table 1, where - means ignore, the current read and write input and output errors are ignored at this time, that is, there are errors in the read and write data and the generation information of the current node read and written in the read and write statistical data is the same as the generation information of the second node stored on the heartbeat disk, thereby avoiding interference from abnormal input and output data.

[0040] In this embodiment, the reading of the slot data of the second node in the shared cluster, screening and saving the valid generation information of the second node according to the root field in the slot data of the second node, and counting the number of consecutive read / write input / output exceptions include: The slot data is set to further include a first root field, a second root field, a third root field and a fourth root field, the initial value of the slot data is zero, the first root field is used to record the valid generation information of the corresponding node, the second root field is used to record the number of consecutive identical generation information read from the corresponding node, the third root field is used to record the number of consecutive different generation information of each node, and the fourth root field is used to record whether the generation information of each node has been authenticated as legal; In response to reading the slot data of the second node, obtaining the value of the fourth root field of the second node, if the value of the fourth root field of the second node is zero, determining that the first node does not store the valid generation information of the second node, and if the value of the fourth root field of the second node is not zero, determining that the first node has stored the valid generation information of the second node; In response to the first node not storing the valid generation information of the second node, obtaining the value of the second root field of the second node, and writing the node generation information corresponding to the normal read and write input and output into the first root field as the valid generation information of the second node by judging whether the second root field of the second node is greater than a third threshold; In response to the first node having saved the generation information of the second node, the value of the third root field of the second node is obtained, and the number of consecutive read and write input and output exceptions in the slot data of the second node is set by determining whether the third root field of the second node is greater than a second threshold.

[0041] Specifically, Figure 3 As shown, in fact, the distributed storage file system (OCFS2) will periodically check the node slot data to obtain the status of each node. Here, checking the node slot data once is used as an example to illustrate.

[0042] First, the node updates its own slot data, then reads the slot data of other nodes, and then verifies the slot data of other nodes in turn. If the gen field in the slot continuously exceeds the first threshold, the node is considered unavailable and is kicked out of the cluster. Otherwise, it determines whether the slot data of other nodes have been verified. If not, it continues to verify the gen field in the slot data of the next node, otherwise the disk heartbeat ends.

[0043] The first root field, the second root field, the third root field and the fourth root field are gen fields. The first root field is gen_record, the second root field is gen_equal, the third root field is gen_change, and the fourth root field is gen_valid.

[0044] like Figure 4 As shown in the figure, M groups of gen fields are added to the memory to temporarily store the status information of the gen fields in the node slot data. Here, M represents the maximum number of nodes supported in the ocfs2 cluster. When the number of nodes is 16, M is 15, and the slot data of each node are slot_0, slot_1, ..., slot_16.

[0045] In this embodiment, the fourth root field is used to record whether the generation information of each node has been authenticated as legal, including: Setting the fourth root field to zero or one; In response to obtaining that the fourth root field is one, determining that the generation information of the corresponding node has been authenticated as legal; In response to obtaining that the fourth root field is zero, it is determined that the generation information of the corresponding node is not authenticated as legal.

[0046] In conjunction with Table 1, in Table 1, when the fourth root field is zero, the generation information of the corresponding node is not authenticated as legal and false is displayed; when the fourth root field is one, the generation information of the corresponding node is authenticated as legal and true is displayed.

[0047] like Figure 5 As shown, in this embodiment, the step of judging whether the second root field of the second node is greater than a third threshold value to write the node generation information corresponding to normal read and write input and output into the first root field as the valid generation information of the second node includes: Determine whether the value of the second root field of the second node is greater than a third threshold, if so, set the fourth root field to one, if not, determine whether the second root field of the second node is zero; If the second root field of the second node is zero, the generation information of the second node is read from the heartbeat disk and stored in the first root field, and the second root field is incremented by one; If the second root field of the second node is not zero, indicating that the generation information of the second node has been stored in the first root field, then determine whether the value of the first root field of the second node in the slot data is the same as the generation information of the second node read from the heartbeat disk, if they are the same, add one to the second root field, if they are different, save the generation information of the second node read from the heartbeat disk to the first root field, and set the second root field to one; If the value of the second root field of the second node plus one is greater than the third threshold, the fourth root field is set to one, indicating that the generation information of the second node has been authenticated as legal.

[0048] like Figure 5 As shown, in this embodiment, when the value of the second root field of the second node is less than or equal to the third threshold, before determining whether the second root field of the second node is zero, the method further includes: Determine whether the generation information of the second node read from the heartbeat disk is zero; If it is zero, end; if it is not zero, determine whether the second root field of the second node is zero.

[0049] Among them, judging whether the generation information of the second node read from the heartbeat disk is zero can avoid misjudgment when the generation information of the second node is zero, and can distinguish the information of generation information being zero from the initially set zero value.

[0050] It is understandable that in order to avoid confusion between the generation information of a node being zero and the initially set zero value, the present application may further set the generation information of each node to a non-zero value.

[0051] Specifically, the distributed storage file system fault tolerance method further includes: Sort all nodes in the shared cluster connected to the heartbeat disk and obtain the sequence number of each node; Set the generation information of each node to a non-zero value; According to the serial number of each node, a plurality of slots are sequentially arranged on the heartbeat disk, and slot data corresponding to a node is stored in each slot, wherein the number of the slots is equal to the number of the nodes; In response to the existence of storage data abnormality in the slot data of the slot, a new slot is created on the heartbeat disk to replace the slot with the storage data abnormality; In response to the target node being offline, clearing or deleting slot data of the slot corresponding to the target node on the heartbeat disk; In response to adding a new node in the shared cluster, a new slot is created on the heartbeat disk for storing slot data of the new node.

[0052] Among them, the number of slots set by the heartbeat disk is equal to the number of nodes. When the number of all nodes in the shared cluster is M, the number of slots is M, which realizes the synchronous management of slot data and corresponding nodes, avoiding the heartbeat disk from generating a large amount of invalid data. At the same time, setting the generation information of each node to a non-zero value can avoid confusion with the initial zero value caused by whether the node's generation information is zero.

[0053] like Figure 5 As shown, check the gen_valid field. If it is true, it means that the generation field of the slot has been successfully temporarily stored, and exit directly. Otherwise, check whether gen_equal is greater than the third threshold (indicates how many times the same generation batch value is read continuously before it is considered legal. It is configured when ocfs2 is started. The value is required to be greater than 1. The purpose of configuring the third threshold is to de-bounce. Only when the generation value is stable, it is considered to be a legal value.). If gen_equal is greater than STABLE_CNT, the generation value is considered legal, and gen_valid is set to true. Otherwise, determine whether the generation value gen_ondisk read from the heartbeat disk is 0. If it is zero, exit. Otherwise, further check the temporarily stored gen_equal value. If gen_equal is 0, it means that the generation value has not yet been temporarily stored. The gen_ondisk read from the heartbeat disk is temporarily stored in gen_record, and the gen_equal count is increased by 1, indicating the number of valid times of the generation value. If the generation value (gen_equal) is not 0, it means that the generation value has been temporarily stored, but it is not considered to be a legal value. It is necessary to further compare gen_ondisk and gen_record. If the two are equal, the gen_equal count value is increased by 1. Otherwise, the latest generation value gen_ondisk read from the heartbeat disk overwrites the content in gen_record, and the gen_equal count is increased by 1.

[0054] like Figure 6 As shown, in this embodiment, the method of setting the number of consecutive read / write input / output exceptions in the slot data of the second node by judging whether the third root field of the second node is greater than a second threshold value includes: If the value of the fourth root field of the second node is one, determine whether the value of the first root field of the second node in the slot data is the same as the generation information of the second node read from the heartbeat disk, and simultaneously determine whether the value of the third root field of the second node is zero; If the value of the first root field of the second node in the slot data is the same as the generation information of the second node read from the heartbeat disk, and the value of the third root field of the second node is not zero and is less than the second threshold, then the third root field is set to zero; If the value of the first root field of the second node in the slot data is different from the generation information of the second node read from the heartbeat disk, the third root field count is increased by one, and it is determined whether the third root field after increase is greater than the second threshold. If so, the number of consecutive read / write input / output exceptions in the slot data of the second node is increased by one, otherwise the process ends.

[0055] like Figure 6 As shown, this solution can tolerate N consecutive (first threshold) heartbeat IO errors. For discontinuous heartbeat IO errors (less than N times), once a normal heartbeat IO is read, the heartbeat IO error count needs to be cleared to prevent the accumulation of heartbeat IO errors. Checking the gen process involves the following judgments: judging whether the gen_valid field is true, which means that the data temporarily stored in gen_record is valid; judging whether the generation value gen_ondisk read from the heartbeat disk is consistent with the temporarily stored gen_record, which means that the current heartbeat IO is normal; judging whether the temporarily stored gen_change is 0, which means that the previous heartbeat IOs are normal, that is, the gen check has passed.

[0056] If gen_valid is true, gen_ondisk and gen_record are consistent, and gen_change is not equal to 0, it means that the temporarily stored generation value is valid, the current heartbeat IO is normal, and the previous heartbeat IO is also normal, then the gen_change count is cleared to 0 and the gen check process ends.

[0057] Furthermore, if gen_ondisk and gen_record are consistent, it means that this heartbeat IO is normal, and this gen check process ends. Otherwise, it is considered that this heartbeat IO is abnormal, and the gen_change count is increased by 1. Then determine whether gen_change is greater than the first threshold (the number of heartbeat IO errors that can be tolerated, corresponding to 'N'). If it is greater, it means that the number of consecutive heartbeat IO errors has reached the maximum tolerance number, then gen_record, gen_change, gen_equal are cleared to 0, gen_valid is set to false, and the node offline event is triggered at the same time, and this gen check ends.

[0058] like Figure 2 As shown, in this embodiment, when controlling the second node to go offline, it also includes: Step S4: clear the slot data of the second node to trigger a command to kick out or isolate the second node from the shared cluster.

[0059] The slot data of the second node is cleared, and the purpose of resetting gen_record and gen_valid is to be compatible with the scenario where the node is back online, because when the node is back online, a new generation information (generation) value will be regenerated.

[0060] like Figure 2 As shown, in this embodiment, the distributed storage file system fault tolerance method also includes: Step S5, before the heartbeat disk performs the next disk heartbeat, determine whether the slot data of all nodes have been verified, if so, end, if not, verify the slot data of the next node.

[0061] Among them, by judging at each node whether the slot data of all other nodes have been verified, interactive checking can be achieved to avoid occasional errors in the reading and writing of multiple nodes themselves, which may cause malfunction of the entire cluster node.

[0062] In this embodiment, the distributed storage file system fault tolerance method further includes: In response to the first node completing the slot data verification of all nodes, the second node is controlled to receive read and write input and output data, the slot data of the second node is updated, and the slot data of the third node in the shared cluster is read and verified until all nodes in the shared cluster connected to the heartbeat disk complete receiving the read and write input and output data.

[0063] Take a two-node cluster consisting of node A and node B, and look at the process of this solution from the perspective of node A as an example.

[0064] Before OCFS2 is started, the first threshold is configured to indicate the number of consecutive IO errors that can be tolerated, the second threshold is configured to indicate the number of consecutive different node generation information set in gen_change that can be tolerated (the number of consecutive times that are inconsistent with the determined node generation information), and the third threshold is configured to indicate how many times the same generation information value is read in gen_equal to be considered legal. Preferably, the total number of cluster nodes is configured to be 16, the first threshold is 3, the second threshold is 2, and the third threshold is 2.

[0065] In this solution, 16 groups of gen-related fields are added to the memory to temporarily store the status information of the gen field in the node slot. In fact, only two groups are used, and the rest are redundant to facilitate subsequent expansion. The gen-related fields include the following four gen_record, gen_equal, gen_change, and gen_valid. gen_record is used to record the valid (i.e. legal) generation information in the node slot, gen_equal is used to record the number of consecutive reads of the same generation in the node slot, gen_change is used to record the number of consecutive reads of illegal (inconsistent with the determined generation information) generation in the node slot, and gen_valid is used to record the legality of the data in the gen_record mentioned above. If it is legal, it is recorded as true, otherwise it is recorded as false. When ocfs2 is started, the above values ​​are all 0.

[0066] First heartbeat.

[0067] First, node A updates its own slot data and writes 0x11111111, then reads the slot data of node B. Here, it is assumed that the generation value in the slot of node B is 0xbbbbbbbb, and then verifies the slot data of node B, which is the following step.

[0068] First, the gen_valid field is checked. At this time, the gen_valid field is 0 (i.e., false), indicating that there is no valid generation field for the slot of node B temporarily stored. Then, gen_equal is checked. At this time, gen_equal is 0, which is less than the third threshold 2. Then, it is further determined that the generation value gen_ondisk read from the heartbeat disk is 0xbbbbbbbb, which is not 0. Then, gen_equal is checked to be 0. At this time, the value 0xbbbbbbbb of gen_ondisk read from the heartbeat disk is temporarily stored in gen_record, i.e., the gen_record field changes from 0 to 0xbbbbbbbb. At the same time, the gen_equal count is increased by 1, i.e., the gen_equal field changes from 0 to 1. All node slots have been verified, and the loop ends. At this time, gen_record is 0xbbbbbbbb, gen_equal is 1, gen_change is 0, and the gen_vaild field is 0.

[0069] Second heartbeat.

[0070] First, node A updates its own slot data and writes 0x11111112, then reads the slot data of node B. Here, it is assumed that the generation value in the slot of node B is still 0xbbbbbbbb. Then, the slot data of node B is verified in the same way as the aforementioned check link. After the loop ends, at this time, gen_record is 0xbbbbbbbb, gen_equal is 2, gen_change is 0, and the gen_vaild field is 0.

[0071] The third heartbeat.

[0072] First, node A updates its own slot data and writes 0x11111113, then reads the slot data of node B. Here, it is assumed that the generation value in the slot of node B is still 0xbbbbbbbb. Then, the slot data of node B is verified in the same way as the aforementioned check link. After the loop ends, at this time, gen_record is 0xbbbbbbbb, gen_equal is 3, gen_change is 0, and the gen_vaild field is 0.

[0073] The fourth heartbeat.

[0074] First, node A updates its own slot data and writes 0x11111114, then reads the slot data of node B. Here, it is assumed that the generation value in the slot of node B is still 0xbbbbbbbb, and then verifies the slot data of node B, which is the following step.

[0075] First, the gen_valid field is checked. At this time, the gen_valid field is 0 (false), indicating that there is no valid generation field for the slot of node B. Then, gen_equal is checked. At this time, gen_equal is 3, which is greater than the third threshold 2. Then gen_valid is set to true, indicating that the value in gen_record is legal. At this time, gen_record is 0xbbbbbbbb, gen_equal is 3, gen_change is 0, and the gen_vaild field is 1.

[0076] In the first three heartbeats, since gen_valid is false, the check gen process will not be executed (i.e. Figure 5 process), but starting from the fourth heartbeat, the gen_valid field is 1 (that is, true), recording the process of valid gen (that is, Figure 4 process) is no longer performed, but the gen process (i.e. Figure 5) will be executed, from then on, heartbeat can effectively perform IO fault tolerance.

[0077] Here, it is assumed that the Xth heartbeat IO (any heartbeat during the normal operation of ocfs2) is abnormal, and the generation value read in the slot of node B becomes 0. The (X+1)th heartbeat IO returns to normal, and the generation value read in the slot of node B is 0xbbbbbbbb, which is the scenario of case 3 in Table 1.

[0078] Xth heartbeat.

[0079] After checking, it is found that gen_valid is true, and gen_ondisk read from the heartbeat disk is 0, which is inconsistent with the gen_record that has been effectively stored. The gen_change count is increased by 1. At this time, the gen_change count value is 1, which is not greater than the first threshold MAX_TOLERATE, and will not trigger the node offline time. The heartbeat IO is effectively fault-tolerant without affecting the cluster function.

[0080] The (X+1)th heartbeat.

[0081] After checking, it is found that gen_valid is true, and the gen_ondisk read from the heartbeat disk is 0xbbbbbbbb, which is consistent with the gen_record that has been effectively stored, and gen_change is not 0, so the gen_change count value is cleared to 0, and the gen check process ends.

[0082] Through the description of this embodiment, it can be found that this solution can effectively tolerate heartbeat IO errors.

[0083] In the above-mentioned distributed storage file system fault-tolerant method, all nodes in the shared cluster connected to the heartbeat disk are used to interactively check whether the read and write input and output data are abnormal, and it is set that the corresponding node is controlled offline only when the number of consecutive read and write input and output abnormalities is greater than the first threshold, thereby increasing the input and output fault-tolerant mechanism and avoiding the problem of occasional input and output errors causing the entire distributed storage file system cluster to be unavailable.

[0084] It should be understood that although Figure 2-Figure 6 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-Figure 6At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0085] In one embodiment, Figure 7 As shown, a distributed storage file system fault tolerance device 10 is provided, comprising: a node slot data management module 1, a slot data verification module 2, and an abnormal node offline control module 3.

[0086] The node slot data management module 1 is used to obtain all nodes in the shared cluster connected to the heartbeat disk, and store the read and write statistics of each node in each disk heartbeat as slot data on the heartbeat disk. The read and write statistics include normal or abnormal read and write input and output, and the number of consecutive read and write input and output abnormalities.

[0087] The slot data verification module 2 is used to update the slot data of the first node in response to the first node receiving read and write input and output data during each disk heartbeat of the heartbeat disk, read the slot data of the second node in the shared cluster, filter and save the valid generation information of the second node according to the root field in the slot data of the second node, and count the number of consecutive read and write input and output anomalies.

[0088] The abnormal node offline control module 3 is used to determine whether the number of consecutive read / write input / output abnormalities in the slot data of the second node is greater than a first threshold. If so, the second node is controlled to be offline. Otherwise, the slot data of the next node is checked.

[0089] In this embodiment, the storage of the read and write statistics of each node in each disk heartbeat on the heartbeat disk through slot data includes: Setting the initial value of the number of consecutive read / write input / output exceptions to zero; Determine whether the read and write input and output of the current node is normal or abnormal during the current disk heartbeat; In response to the current node reading and writing input and output being normal during the current disk heartbeat, determining whether the current node reading and writing input and output is normal or abnormal during the next heartbeat reading and writing; In response to a read / write input / output exception of the current node during a current disk heartbeat, the number of consecutive read / write input / output exceptions of the current node is increased by one.

[0090] In this embodiment, determining whether the read / write input / output of the current node is normal or abnormal during the current disk heartbeat includes: The generation information of each node is stored on the heartbeat disk, and the read and write statistics also include the generation information of the current node performing the reading and writing; Read the heartbeat read and write statistics of the second node that shares a connection with the first node, and determine whether the generation information of the current node performing reading and writing in the read and write statistics is the same as the generation information of the second node stored on the heartbeat disk; if they are different, determine that the current read and write input and output are abnormal; if they are the same, determine that the current read and write input and output are normal.

[0091] In this embodiment, determining whether the read / write input / output of the current node is normal or abnormal during the current disk heartbeat also includes: Determine whether there is an error in the data read and written in the read and write statistical data; If there is no error and the generation information of the current node being read and written in the read and write statistical data is the same as the generation information of the second node stored on the heartbeat disk, it is determined that the current read and write input and output are normal; If there is an error and the generation information of the current node being read and written in the read and write statistical data is the same as the generation information of the second node stored on the heartbeat disk, the current read and write input and output error is ignored.

[0092] In this embodiment, the reading of the slot data of the second node in the shared cluster, screening and saving the valid generation information of the second node according to the root field in the slot data of the second node, and counting the number of consecutive read / write input / output exceptions include: The slot data is set to further include a first root field, a second root field, a third root field and a fourth root field, the initial value of the slot data is zero, the first root field is used to record the valid generation information of the corresponding node, the second root field is used to record the number of consecutive identical generation information read from the corresponding node, the third root field is used to record the number of consecutive different generation information of each node, and the fourth root field is used to record whether the generation information of each node has been authenticated as legal; In response to reading the slot data of the second node, obtaining the value of the fourth root field of the second node, if the value of the fourth root field of the second node is zero, determining that the first node does not store the valid generation information of the second node, and if the value of the fourth root field of the second node is not zero, determining that the first node has stored the valid generation information of the second node; In response to the first node not storing the valid generation information of the second node, obtaining the value of the second root field of the second node, and writing the node generation information corresponding to the normal read and write input and output into the first root field as the valid generation information of the second node by judging whether the second root field of the second node is greater than a third threshold; In response to the first node having saved the generation information of the second node, the value of the third root field of the second node is obtained, and the number of consecutive read and write input and output exceptions in the slot data of the second node is set by determining whether the third root field of the second node is greater than a second threshold.

[0093] In this embodiment, the fourth root field is used to record whether the generation information of each node has been authenticated as legal, including: Setting the fourth root field to zero or one; In response to obtaining that the fourth root field is one, determining that the generation information of the corresponding node has been authenticated as legal; In response to obtaining that the fourth root field is zero, it is determined that the generation information of the corresponding node is not authenticated as legal.

[0094] In this embodiment, judging whether the second root field of the second node is greater than a third threshold value to write the node generation information corresponding to normal read and write input and output into the first root field as the valid generation information of the second node includes: Determine whether the value of the second root field of the second node is greater than a third threshold, if so, set the fourth root field to one, if not, determine whether the second root field of the second node is zero; If the second root field of the second node is zero, the generation information of the second node is read from the heartbeat disk and stored in the first root field, and the second root field is incremented by one; If the second root field of the second node is not zero, indicating that the generation information of the second node has been stored in the first root field, then determine whether the value of the first root field of the second node in the slot data is the same as the generation information of the second node read from the heartbeat disk, if they are the same, add one to the second root field, if they are different, save the generation information of the second node read from the heartbeat disk to the first root field, and set the second root field to one; If the value of the second root field of the second node plus one is greater than the third threshold, the fourth root field is set to one, indicating that the generation information of the second node has been authenticated as legal.

[0095] In this embodiment, when the value of the second root field of the second node is less than or equal to the third threshold, before determining whether the second root field of the second node is zero, the method further includes: Determine whether the generation information of the second node read from the heartbeat disk is zero; If it is zero, end; if it is not zero, determine whether the second root field of the second node is zero.

[0096] In this embodiment, the step of setting the number of consecutive read / write input / output exceptions in the slot data of the second node by judging whether the third root field of the second node is greater than a second threshold value includes: If the value of the fourth root field of the second node is one, determine whether the value of the first root field of the second node in the slot data is the same as the generation information of the second node read from the heartbeat disk, and simultaneously determine whether the value of the third root field of the second node is zero; If the value of the first root field of the second node in the slot data is the same as the generation information of the second node read from the heartbeat disk, and the value of the third root field of the second node is not zero and is less than the second threshold, then the third root field is set to zero; If the value of the first root field of the second node in the slot data is different from the generation information of the second node read from the heartbeat disk, the third root field count is increased by one, and it is determined whether the third root field after increase is greater than the second threshold. If so, the number of consecutive read / write input / output exceptions in the slot data of the second node is increased by one, otherwise the process ends.

[0097] In this embodiment, if Figure 7 As shown, the distributed storage file system fault-tolerant device 10 further includes: a slot data clearing module 4 .

[0098] When controlling the second node to go offline, the slot data clearing module 4 is used to clear the slot data of the second node and trigger a command to kick out or isolate the second node from the shared cluster.

[0099] In this embodiment, if Figure 7 As shown, the distributed storage file system fault tolerance device 10 also includes: an all-node verification management module 5 .

[0100] The all-node verification management module 5 is used to determine whether the slot data of all nodes have been verified before the heartbeat disk performs the next disk heartbeat. If so, the process ends; otherwise, the slot data of the next node is verified.

[0101] In this embodiment, if Figure 7 As shown, the distributed storage file system fault tolerance device 10 also includes: a cyclic verification control module 6.

[0102] In response to the first node completing the slot data verification of all nodes, the cyclic verification control module 6 controls the second node to receive read and write input and output data, updates the slot data of the second node, reads and verifies the slot data of the third node in the shared cluster, until all nodes in the shared cluster connected to the heartbeat disk complete receiving the read and write input and output data.

[0103] In the above-mentioned distributed storage file system fault-tolerant device, all nodes in the shared cluster connected to the heartbeat disk are used to interactively check whether the read and write input and output data are abnormal, and it is set that the corresponding node is controlled offline only when the number of consecutive read and write input and output abnormalities is greater than the first threshold, thereby increasing the input and output fault-tolerant mechanism and avoiding the problem of occasional input and output errors causing the entire distributed storage file system cluster to be unavailable.

[0104] For the specific definition of the distributed storage file system fault-tolerant device, please refer to the definition of the distributed storage file system fault-tolerant method above, which will not be repeated here. Each module in the above-mentioned distributed storage file system fault-tolerant device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0105] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store distributed storage file system fault-tolerant data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a distributed storage file system fault-tolerant method is implemented.

[0106] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0107] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps: Obtain all nodes in the shared cluster connected to the heartbeat disk, and store the read and write statistics of each node in each disk heartbeat as slot data on the heartbeat disk, wherein the read and write statistics include whether the read and write input and output are normal or abnormal, and the number of consecutive read and write input and output abnormalities; At each disk heartbeat of the heartbeat disk, in response to the first node receiving read / write input / output data, updating the slot data of the first node, reading the slot data of the second node in the shared cluster, filtering and saving the valid generation information of the second node according to the root field in the slot data of the second node, and counting the number of consecutive read / write input / output anomalies; Determine whether the number of consecutive read / write input / output anomalies in the slot data of the second node is greater than a first threshold. If so, control the second node to go offline. Otherwise, check the slot data of the next node.

[0108] For specific limitations on the steps implemented when the computer program is executed by the processor, please refer to the limitations on the method for fault tolerance of the distributed storage file system above, which will not be repeated here.

[0109] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: Obtain all nodes in the shared cluster connected to the heartbeat disk, and store the read and write statistics of each node in each disk heartbeat as slot data on the heartbeat disk, wherein the read and write statistics include whether the read and write input and output are normal or abnormal, and the number of consecutive read and write input and output abnormalities; At each disk heartbeat of the heartbeat disk, in response to the first node receiving read / write input / output data, updating the slot data of the first node, reading the slot data of the second node in the shared cluster, filtering and saving the valid generation information of the second node according to the root field in the slot data of the second node, and counting the number of consecutive read / write input / output anomalies; Determine whether the number of consecutive read / write input / output anomalies in the slot data of the second node is greater than a first threshold. If so, control the second node to go offline. Otherwise, check the slot data of the next node.

[0110] For specific limitations on the steps implemented when the processor executes the computer program, please refer to the limitations on the method for fault tolerance of the distributed storage file system above, which will not be repeated here.

[0111] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: Obtain all nodes in the shared cluster connected to the heartbeat disk, and store the read and write statistics of each node in each disk heartbeat as slot data on the heartbeat disk, wherein the read and write statistics include whether the read and write input and output are normal or abnormal, and the number of consecutive read and write input and output abnormalities; At each disk heartbeat of the heartbeat disk, in response to the first node receiving read / write input / output data, updating the slot data of the first node, reading the slot data of the second node in the shared cluster, filtering and saving the valid generation information of the second node according to the root field in the slot data of the second node, and counting the number of consecutive read / write input / output anomalies; Determine whether the number of consecutive read / write input / output anomalies in the slot data of the second node is greater than a first threshold. If so, control the second node to go offline. Otherwise, check the slot data of the next node.

[0112] For specific limitations on the steps implemented when the computer program is executed by the processor, please refer to the limitations on the method for fault tolerance of the distributed storage file system above, which will not be repeated here.

[0113] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0114] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0115] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. A distributed storage file system fault tolerance method, characterized in that: include: Obtain all nodes in the shared cluster connected to the heartbeat disk, and store the read and write statistics of each node in each disk heartbeat as slot data on the heartbeat disk, wherein the read and write statistics include whether the read and write input and output are normal or abnormal, and the number of consecutive read and write input and output abnormalities; At each disk heartbeat of the heartbeat disk, in response to the first node receiving read / write input / output data, updating the slot data of the first node, reading the slot data of the second node in the shared cluster, filtering and saving the valid generation information of the second node according to the root field in the slot data of the second node, and counting the number of consecutive read / write input / output anomalies; Determine whether the number of consecutive read / write input / output anomalies in the slot data of the second node is greater than a first threshold. If so, control the second node to go offline. Otherwise, check the slot data of the next node.

2. The distributed storage file system fault tolerance method according to claim 1, characterized in that: The storage of read and write statistics of each node in each disk heartbeat through slot data on the heartbeat disk includes: Setting the initial value of the number of consecutive read / write input / output exceptions to zero; Determine whether the read and write input and output of the current node is normal or abnormal during the current disk heartbeat; In response to the current node reading and writing input and output being normal during the current disk heartbeat, determining whether the current node reading and writing input and output is normal or abnormal during the next heartbeat reading and writing; In response to a read / write input / output exception of the current node during a current disk heartbeat, the number of consecutive read / write input / output exceptions of the current node is increased by one.

3. The distributed storage file system fault tolerance method according to claim 2, characterized in that: The determination of whether the read / write input / output of the current node is normal or abnormal during the current disk heartbeat includes: The generation information of each node is stored on the heartbeat disk, and the read and write statistics also include the generation information of the current node performing the reading and writing; Read the heartbeat read and write statistics of the second node that shares a connection with the first node, and determine whether the generation information of the current node performing reading and writing in the read and write statistics is the same as the generation information of the second node stored on the heartbeat disk; if they are different, determine that the current read and write input and output are abnormal; if they are the same, determine that the current read and write input and output are normal.

4. The distributed storage file system fault tolerance method according to claim 3, characterized in that: The determining whether the read / write input / output of the current node is normal or abnormal during the current disk heartbeat also includes: Determine whether there is an error in the data read and written in the read and write statistical data; If there is no error and the generation information of the current node being read and written in the read and write statistical data is the same as the generation information of the second node stored on the heartbeat disk, it is determined that the current read and write input and output are normal; If there is an error and the generation information of the current node being read and written in the read and write statistical data is the same as the generation information of the second node stored on the heartbeat disk, the current read and write input and output error is ignored.

5. The distributed storage file system fault tolerance method according to claim 3, characterized in that: The reading of the slot data of the second node in the shared cluster, screening and saving the valid generation information of the second node according to the root field in the slot data of the second node, and counting the number of consecutive occurrences of read / write input / output anomalies includes: The slot data is set to further include a first root field, a second root field, a third root field and a fourth root field, the initial value of the slot data is zero, the first root field is used to record the valid generation information of the corresponding node, the second root field is used to record the number of consecutive identical generation information read from the corresponding node, the third root field is used to record the number of consecutive different generation information of each node, and the fourth root field is used to record whether the generation information of each node has been authenticated as legal; In response to reading the slot data of the second node, obtaining the value of the fourth root field of the second node, if the value of the fourth root field of the second node is zero, determining that the first node does not store the valid generation information of the second node, and if the value of the fourth root field of the second node is not zero, determining that the first node has stored the valid generation information of the second node; In response to the first node not storing the valid generation information of the second node, obtaining the value of the second root field of the second node, and writing the node generation information corresponding to the normal read and write input and output into the first root field as the valid generation information of the second node by judging whether the second root field of the second node is greater than a third threshold; In response to the first node having saved the generation information of the second node, the value of the third root field of the second node is obtained, and the number of consecutive read and write input and output exceptions in the slot data of the second node is set by determining whether the third root field of the second node is greater than a second threshold.

6. The distributed storage file system fault tolerance method according to claim 5, characterized in that: The fourth root field is used to record whether the generation information of each node has been authenticated as legal, including: Setting the fourth root field to zero or one; In response to obtaining that the fourth root field is one, determining that the generation information of the corresponding node has been authenticated as legal; In response to obtaining that the fourth root field is zero, it is determined that the generation information of the corresponding node is not authenticated as legal.

7. The distributed storage file system fault tolerance method according to claim 5, characterized in that: The step of judging whether the second root field of the second node is greater than a third threshold value to write the node generation information corresponding to the normal read / write input and output into the first root field as the valid generation information of the second node comprises: Determine whether the value of the second root field of the second node is greater than a third threshold, if so, set the fourth root field to one, if not, determine whether the second root field of the second node is zero; If the second root field of the second node is zero, the generation information of the second node is read from the heartbeat disk and stored in the first root field, and the second root field is incremented by one; If the second root field of the second node is not zero, indicating that the generation information of the second node has been stored in the first root field, then determine whether the value of the first root field of the second node in the slot data is the same as the generation information of the second node read from the heartbeat disk, if they are the same, add one to the second root field, if they are different, save the generation information of the second node read from the heartbeat disk to the first root field, and set the second root field to one; If the value of the second root field of the second node plus one is greater than the third threshold, the fourth root field is set to one.

8. The distributed storage file system fault tolerance method according to claim 7, characterized in that: When the value of the second root field of the second node is less than or equal to a third threshold, before determining whether the second root field of the second node is zero, the method further includes: Determine whether the generation information of the second node read from the heartbeat disk is zero; If it is zero, end; if it is not zero, determine whether the second root field of the second node is zero.

9. The distributed storage file system fault tolerance method according to claim 5, characterized in that: The step of setting the number of consecutive read / write input / output exceptions in the slot data of the second node by judging whether the third root field of the second node is greater than a second threshold value comprises: If the value of the fourth root field of the second node is one, determine whether the value of the first root field of the second node in the slot data is the same as the generation information of the second node read from the heartbeat disk, and simultaneously determine whether the value of the third root field of the second node is zero; If the value of the first root field of the second node in the slot data is the same as the generation information of the second node read from the heartbeat disk, and the value of the third root field of the second node is not zero and is less than the second threshold, then the third root field is set to zero; If the value of the first root field of the second node in the slot data is different from the generation information of the second node read from the heartbeat disk, the third root field count is increased by one, and it is determined whether the third root field after increase is greater than the second threshold. If so, the number of consecutive read / write input / output exceptions in the slot data of the second node is increased by one, otherwise the process ends.

10. The distributed storage file system fault tolerance method according to claim 1, characterized in that: When controlling the second node to go offline, the method further includes: The slot data of the second node is cleared to trigger a command to kick out or isolate the second node from the shared cluster.

11. The distributed storage file system fault tolerance method according to claim 1, characterized in that: The method further comprises: Before the heartbeat disk performs the next disk heartbeat, it is determined whether the slot data of all nodes have been verified, and if so, the process ends; otherwise, the slot data of the next node is verified.

12. The distributed storage file system fault tolerance method according to claim 11, characterized in that: The method further comprises: In response to the first node completing the slot data verification of all nodes, the second node is controlled to receive read and write input and output data, the slot data of the second node is updated, and the slot data of the third node in the shared cluster is read and verified until all nodes in the shared cluster connected to the heartbeat disk complete receiving the read and write input and output data.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

14. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 12 are implemented.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

Images