A method and product for handling request timeouts

Abstract

The embodiment of the present application provides a request timeout processing method and product, the method comprises that each object storage process in the distributed storage cluster monitors whether the request processing time of itself exists timeout, the object storage process existing timeout reports a request timeout message to a cluster monitoring process of the distributed storage cluster, the request timeout message reported by the object storage process existing timeout comprises a target object storage process causing the object storage process to exist timeout; the cluster monitoring process determines whether the target object storage process is an abnormal object storage process according to a disk state corresponding to the target object storage process and a node state where the target object storage process is located, and executes a timeout isolation strategy on the abnormal object storage process. The embodiment of the present application aims at reducing the business influence on the distributed storage cluster.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of data storage, and in particular, to a request timeout processing method and product. BACKGROUND

[0002] In the normal business read-write request processing process of a distributed storage cluster, abnormal phenomena such as "slow disk" may occur due to network fluctuations, node resource exhaustion, and disk failures. These abnormal scenarios can affect part of the OSD processes of the distributed storage cluster, and further affect the processing of IO.

[0003] However, due to the need for redundancy to protect data security in the distributed storage cluster, a storage group PG corresponding to the data of a request usually includes multiple OSD processes. The OSD process members of each PG need to interact with each other. Therefore, if the IO processing of part of the OSD processes of the distributed storage cluster becomes slow or even blocked, the processing flow of other OSDs will also be affected, eventually leading to slow op in the distributed storage cluster, or even causing the business of the distributed storage cluster to be interrupted.

[0004] Currently, when slow op occurs in the distributed storage cluster, there is no standard processing flow to quickly handle it to allow the distributed storage cluster to adaptively recover from op blocking state to normal. Because the reasons for causing slow op in the distributed storage cluster are very complex, the current conventional processing method is to manually analyze and manually handle the corresponding problems according to the detailed problem reasons to help the distributed storage cluster recover to normal.

[0005] However, in the manual analysis mode, the process from discovering the problem to human intervention and finally solving the problem will result in high labor cost and long time consumption, and the business of the distributed storage cluster will be affected for a long time. SUMMARY

[0006] Embodiments of the present application provide a request timeout processing method and product, aiming to reduce the impact on the business of the distributed storage cluster.

[0007] In a first aspect, embodiments of the present application provide a request timeout processing method applied to a distributed storage cluster, the method comprising:

[0008] Each object storage process in the distributed storage cluster monitors whether the request processing time of itself is timed out. The object storage process that is timed out reports a request timeout message to a cluster monitoring process of the distributed storage cluster. The request timeout message reported by the object storage process that is timed out includes a target object storage process that causes the object storage process to be timed out.

[0009] The cluster monitoring process determines whether the target object storage process is an abnormal object storage process according to the disk status corresponding to the target object storage process and the node status where the target object storage process is located, and executes a timeout isolation strategy for the abnormal object storage process.

[0010] Optionally, each object storage process in the distributed storage cluster monitors whether its own request processing time has timed out, and the object storage process with timeout reports a request timeout message to the cluster monitoring process of the distributed storage cluster, including:

[0011] Each object storage process in the distributed storage cluster monitors its own request processing time at a predetermined time interval to determine whether it is greater than or equal to a timeout threshold. If the request processing time is greater than or equal to the timeout threshold, a timeout problem occurs.

[0012] The object storage process with timeout determines whether a request timeout message corresponding to the current request has been reported;

[0013] When the request timeout message has not been reported, or the time interval since the last reporting of the request timeout message exceeds the target reporting interval, the object storage process reports the request timeout message to the cluster monitoring process of the distributed storage cluster.

[0014] Optionally, after the object storage process reports the request timeout message to the cluster monitoring process of the distributed storage cluster, the method further includes:

[0015] The object storage process detects whether the request with the timeout problem has been processed;

[0016] When the request with the timeout problem is processed, the object storage process reports a cancellation request corresponding to the request timeout message to the cluster monitoring process;

[0017] In response to the cancellation request, the cluster monitoring process deletes the request timeout message of the object storage process stored in the cluster monitoring process.

[0018] Optionally, the cluster monitoring process determines whether the target object storage process is an abnormal object storage process according to the disk status corresponding to the target object storage process and the node status where the target object storage process is located, including:

[0019] The cluster monitoring process determines whether the disk statuses corresponding to the target object storage processes are all in slow disk status;

[0020] When all the disks corresponding to the target object storage processes are in slow disk state, all the target object storage processes are regarded as abnormal object storage processes;

[0021] When all disks corresponding to the target object storage processes are not all in slow disk state, determine whether the target object storage processes that are not in slow disk state are concentrated in the target number of nodes;

[0022] If the target object storage processes that are not in a slow disk state are not concentrated on the target number of nodes, it indicates that the distributed storage cluster is under temporary business processing pressure, and the timeout processing process is skipped;

[0023] When the target object storage processes that are not in the slow disk state are concentrated on the target number of nodes, determine whether the node status of each node is in the network sub-health state;

[0024] When the node is in a sub-healthy network state, the target object storage process on the node is treated as an abnormal object storage process;

[0025] When the node is not in a network sub-health state and the node state of the node is in a node sub-health state, the target object storage process on the node is regarded as an abnormal object storage process.

[0026] Optionally, when the target object storage processes that are not in a slow disk state are concentrated on a target number of nodes, after determining whether each node is in a network sub-health state, the method further includes:

[0027] When the node is not in a network sub-health state and the node is not in a node sub-health state, the cluster monitoring process reports an unknown timeout prompt message.

[0028] Optionally, executing a timeout isolation strategy for the abnormal object storage process includes:

[0029] For each abnormal object storage process, determine whether the storage pool corresponding to the abnormal object storage process is in a state close to the super fault domain;

[0030] When the storage pool corresponding to the abnormal object storage process is not in a state close to the super fault domain, isolate the abnormal object storage process;

[0031] When the storage pool corresponding to the abnormal object storage process is in a state close to exceeding the fault domain, the abnormal object storage process is not isolated.

[0032] Optionally, for each abnormal object storage process, determining whether a storage pool corresponding to the abnormal object storage process is in a state close to exceeding the fault domain includes:

[0033] For each abnormal object storage process, traverse the storage group corresponding to the abnormal object storage process;

[0034] When at least one storage group corresponding to the abnormal object storage process is in a degraded state and at least one data in the storage group has no redundancy, it is determined that the storage pool corresponding to the abnormal object storage process is in a state close to a super fault domain.

[0035] In a second aspect, an embodiment of the present application provides a request timeout processing device, which is applied to a distributed storage cluster, and the device includes:

[0036] A timeout reporting module is configured to monitor whether each object storage process in the distributed storage cluster has timed out its own request processing time, and the object storage process with timeout reports a request timeout message to the cluster monitoring process of the distributed storage cluster, wherein the request timeout message reported by the object storage process with timeout includes the target object storage process that caused the object storage process to time out;

[0037] The timeout isolation module is used for the cluster monitoring process to determine whether the target object storage process is an abnormal object storage process based on the disk status corresponding to the target object storage process and the node status where the target object storage process is located, and to execute the timeout isolation strategy for the abnormal object storage process.

[0038] In a third aspect, an embodiment of the present application provides a computer device comprising: at least one processor, and a memory, wherein the memory stores a computer program that can be run on the processor, wherein when the processor executes the computer program, the request timeout processing method described in the first aspect of the embodiment is executed.

[0039] In a fourth aspect, an embodiment of the present application provides a non-volatile readable storage medium, which stores a computer program, wherein when the computer program is executed by a processor, the request timeout processing method described in the first aspect of the embodiment is executed.

[0040] Beneficial effects:

[0041] In the processing method provided by this embodiment, each object storage process in the distributed storage cluster monitors whether its own request processing time has timed out. The object storage process that has timed out reports a request timeout message to the cluster monitoring process of the distributed storage cluster. The request timeout message reported by the object storage process that has timed out includes the target object storage process that caused the object storage process to time out. The cluster monitoring process determines whether the target object storage process is an abnormal object storage process based on the disk status corresponding to the target object storage process and the node status where the target object storage process is located, and executes a timeout isolation strategy for the abnormal object storage process.

[0042] When one object storage process in the distributed storage monitors the request processing time of itself and a timeout problem occurs, the target object storage process causing the timeout is reported to the cluster monitoring process through a request timeout message, the cluster monitoring process locates the abnormal object storage process according to the disk state of the target object storage process and the node state of each target object storage process, and determines whether to isolate the abnormal object storage process, so that the isolation of the abnormal object storage process can be automatically realized when the IO timeout occurs. Compared with manually processing the IO timeout problem, the timeliness of processing can be significantly improved in the process of processing the IO timeout problem, and the time of the business impact of the distributed storage cluster caused by the IO timeout problem can be greatly reduced. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the embodiments or the related art description will be briefly introduced.

[0044] Figure 1 A step flowchart of the request timeout processing method provided by the embodiments of the present application is shown;

[0045] Figure 2 An execution flowchart of the IO timeout detection module provided by the embodiments of the present application is shown;

[0046] Figure 3 An execution flowchart of the IO timeout problem locating module provided by the embodiments of the present application is shown;

[0047] Figure 4 An execution flowchart of the IO timeout fault processing module provided by the embodiments of the present application is shown;

[0048] Figure 5 A functional module diagram of the request timeout processing device provided by the embodiments of the present application is shown;

[0049] Figure 6 A schematic diagram of the computer device provided by the embodiments of the present application is shown;

[0050] Figure 7 A schematic diagram of the non-volatile readable storage medium provided by the embodiments of the present application is shown. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.

[0052] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the various embodiments of the present application will be described in detail below with reference to the drawings. However, those skilled in the art can understand that in the various embodiments of the present application, many technical details are proposed in order to make the readers better understand the present application. However, the technical solutions claimed by the present application can be implemented even without these technical details and various changes and modifications based on the following embodiments. The division of the following various embodiments is for the convenience of description, and should not constitute any limitation on the specific implementation of the present application, and the various embodiments can be combined and quoted with each other without contradiction.

[0053] OSD: Object-based Storage Device, object storage process;

[0054] Monitor: cluster monitoring service;

[0055] Slow Op: slow request (i.e. IO timeout request);

[0056] Op blocking state refers to the state of a process when it is waiting for an event or resource to be ready; when a process requests the use and allocation of a resource, or waits for the occurrence of an event (such as the completion of an I / O operation), it will be transferred from the running state to the blocking state; in the blocking state, even if the processor is allocated to the process, it cannot execute because the process needs to wait for the resource to be ready.

[0057] In the normal business read and write request processing process of the distributed storage cluster, due to network fluctuations, node resource exhaustion and disk failures, abnormal phenomena such as "slow disk" may occur, which will affect part of the OSD processes of the distributed storage cluster, and further affect the processing of IO.

[0058] However, due to the need for redundancy for data security in the distributed storage cluster, the storage group PG corresponding to a request data usually includes multiple OSD processes, and the OSD process members of each PG need to interact, for example, in a 3-copy storage mode, for a PG corresponding to a data, 3 OSD processes will store data in 3 hard disks, including a master OSD process and two slave OSD processes, i.e. the OSD processes need to interact.

[0059] Therefore, if the IO processing of part of the OSD processes of the distributed storage cluster becomes slow or even blocked, it will affect the processing flow of other OSDs, and eventually cause the distributed storage cluster to have a slow request, or even cause the business of the distributed storage cluster to be interrupted.

[0060] Currently, when a slow op occurs in a distributed storage cluster, there is no standard processing procedure to quickly handle it to enable the distributed storage cluster to adaptively recover from the op blocking state. Because the reasons for causing the slow op in the distributed storage cluster are very complex, the current regular processing method is to manually analyze and manually handle the problem according to the detailed problem reason to help the distributed storage cluster recover to normal. However, in the manual analysis mode, the process from discovering the problem to human intervention and finally solving the problem will cause high labor cost and long time consumption, and the business of the distributed storage cluster is affected for a long time.

[0061] Therefore, in order to reduce the impact on the business of the distributed storage cluster, an embodiment of the present application provides a request timeout processing method.

[0062] Referring to Figure 1 , a step flowchart of a request timeout processing method provided by an embodiment of the present application is shown, the method is applied to a distributed storage cluster, and can include the following steps:

[0063] S101: Each object storage process in the distributed storage cluster monitors whether the request processing time of itself exists timeout, and the object storage process existing timeout reports a request timeout message to a cluster monitoring process of the distributed storage cluster, wherein the request timeout message reported by the object storage process existing timeout includes a target object storage process causing the timeout of the object storage process.

[0064] A plurality of object storage processes, i.e., a plurality of OSD processes, are maintained in the distributed storage cluster, and usually one OSD process corresponds to one disk. When the resource is limited, such as disk hardware failure, too high CPU occupation or too high memory occupation, if the IO processing of part of the OSD processes becomes slow or even blocked, the processing procedure of other OSDs will also be affected.

[0065] Therefore, in the embodiment, each object storage process in the distributed storage cluster monitors whether the request processing time of itself exists timeout, and when the object storage process monitors that the request processing time of itself exists timeout, a request timeout message is reported to the cluster monitoring process of the distributed storage cluster, i.e., to the Monitor of the distributed storage cluster.

[0066] In a feasible implementation, the process that each object storage process in the distributed storage cluster monitors whether the request processing time of itself exists timeout, and the object storage process existing timeout reports a request timeout message to the cluster monitoring process of the distributed storage cluster, is as follows:

[0067] A1: Each object storage process in the distributed storage cluster monitors its own request processing time at regular intervals to see whether it is greater than or equal to a timeout threshold.

[0068] For example, in order to avoid the OSD process detecting IO timeouts in real time and reporting them, which would result in excessive processing costs, a timer can be set for the OSD process. The timer duration is configured as a calibration time interval. For example, the calibration time interval can be set to 10s, that is, the OSD process checks whether it has an IO timeout problem every 10s.

[0069] When the OSD process handles the current IO request, it records the request processing time and compares it with the preset timeout threshold to determine whether the OSD process itself has an IO timeout problem; the timeout threshold setting can be customized according to the actual application requirements of the distributed storage cluster, and this embodiment does not impose any restrictions.

[0070] A2: When the request processing time is greater than or equal to the timeout threshold, it indicates that a timeout problem exists.

[0071] When the object storage process monitors that its own request processing time is less than the timeout threshold, it is necessary to determine whether the current request has not been processed after 10 seconds. If not, it is necessary to determine whether it has timed out again based on the request processing time.

[0072] A3: The object storage process with a timeout determines whether a request timeout message corresponding to the current request has been reported.

[0073] Since the cluster monitoring process takes time to locate IO timeouts, if an IO timeout occurs in the OSD process, it can determine whether the request timeout message corresponding to the current request has been reported, and the time interval between the last report and the current time to determine whether to continue reporting the request timeout message corresponding to the current request.

[0074] The request timeout message reported by the OSD process includes the target OSD process that caused the OSD process to time out. For example, if in the 3-copy storage method of data A, the processing process of the master OSD process 1 causes the IO timeout of the slave OSD process 2 and the slave OSD process 3, and then the request timeout messages reported by the slave OSD process 2 and the slave OSD process 3 include the master OSD process 1.

[0075] If the OSD process has not reported a request timeout message corresponding to the current request, the request timeout message can be reported to the cluster monitoring process, namely Monitor.

[0076] A4: When the request timeout message has not been reported, or the time interval since the last reporting of the request timeout message exceeds the target reporting interval, the object storage process reports the request timeout message to the cluster monitoring process of the distributed storage cluster.

[0077] If the OSD process has already reported the request timeout message corresponding to the current request, but the time interval from the last time the request timeout message is reported has exceeded the target reporting interval, the request timeout message can be reported to the Monitor again.

[0078] If the OSD process has already reported the request timeout message corresponding to the current request, but the time interval from the last time the request timeout message is reported is still less than the target reporting interval, the request timeout message is not reported to the Monitor again.

[0079] Specifically, after the object storage process reports the request timeout message to the cluster monitoring process of the distributed storage cluster, the object storage process can further detect whether the request with a timeout problem is processed.

[0080] When the request with a timeout problem is processed, the object storage process reports a cancel request corresponding to the request timeout message to the cluster monitoring process; and the cluster monitoring process deletes the request timeout message of the object storage process stored in the cluster monitoring process in response to the cancel request.

[0081] That is, if the OSD process currently has no timeout request, the request timeout message of the OSD process is reported to the Monitor, and the Monitor deletes the request timeout message of the OSD process from the data structure recorded by the Monitor.

[0082] S102: The cluster monitoring process determines whether a target object storage process is an abnormal object storage process according to a disk state corresponding to the target object storage process and a node state in which the target object storage process is located, and executes a timeout isolation strategy on the abnormal object storage process.

[0083] In an available embodiment, the process in which the cluster monitoring process determines whether a target object storage process is an abnormal object storage process according to a disk state corresponding to the target object storage process and a node state in which the target object storage process is located can include:

[0084] B1: The cluster monitoring process determines whether the disk state corresponding to the target object storage process is all in a slow disk state.

[0085] Specifically, in the embodiment, the disk state includes a slow disk state or a non-slow disk state, wherein the slow disk state refers to a phenomenon that the disk hardware corresponding to the OSD process appears due to damage or failure and the like.

[0086] In the actual implementation process, the cluster monitoring process monitors and obtains the disk state of the target object storage process.

[0087] At the current moment, the number of target OSD processes received by the cluster monitoring process may be only one or more.

[0088] For example, OSD1, OSD2, and OSDn all report OSD0 as the target object storage process that caused their own IO timeouts, and OSD3 and OSD4 report OSD5 as the target object storage process that caused their own IO timeouts. The Monitor first determines whether the disk status of OSD0 and OSD5 are all slow disk status, that is, whether the IO processing timeout is caused by the disk failure corresponding to OSD0 and OSD5.

[0089] For example, OSD1-OSDn all report OSD0 as the target object storage process that causes their own IO timeout. At this time, Monitor only receives OSD0 as the target OSD process, and then determines whether the disk status corresponding to OSD0 is slow disk status.

[0090] In the actual implementation process, it is also possible to detect whether the disk status corresponding to the OSD process with timeout is a slow disk status. For example, if OSD1, OSD2 and OSDn all report OSD0 as the target object storage process that causes their own IO timeout, in addition to judging whether OSD0 is in a slow disk status, it is also possible to judge whether OSD1, OSD2 and OSDn themselves are in a slow disk status, that is, to determine whether the IO timeouts of OSD1, OSD2 and OSDn are due to their own problems rather than OSD0; if the OSD process that reports other OSD processes as causing its own IO timeout is actually due to its own abnormal state of slow disk, the OSD process can be regarded as an abnormal object storage process, and then determine whether to isolate and judge the OSD process to avoid the OSD process from affecting other OSD processes in the future.

[0091] B2: When all disks corresponding to all target object storage processes are in slow disk state, all target object storage processes are treated as abnormal object storage processes.

[0092] For example, when OSD0 and OSD5 are all OSD0 and OSD5, OSD0 and OSD5 are both stored as abnormal objects.

[0093] B3: When not all disks corresponding to all target object storage processes are in slow disk state, determine whether the target object storage processes that are not in slow disk state are concentrated in the target number of nodes.

[0094] Usually, the number of OSDs on each node of a distributed storage cluster and the number between OSDs and PGs are balanced. If the cluster is overloaded due to temporary excessive business pressure, causing the distributed storage cluster to generate slow ops, the OSDs that have timed out by IO will be distributed on each node, and the number distribution will be relatively even. Therefore, based on the distribution of OSDs that have timed out by IO, it can be determined whether the problem is with certain nodes, causing the IO timeout phenomenon.

[0095] For example, the target number can be 1, that is, whether the storage processes of all target objects that are not in a slow disk state are concentrated in one node. The target number can also be set according to the actual application needs, but the configuration of the target number usually does not exceed the data redundancy number of the distributed storage cluster. For example, when the distributed storage cluster adopts a 3-copy method, the target number configuration does not exceed 3.

[0096] B4: When the target object storage processes that are not in the slow disk state are not concentrated on the target number of nodes, it indicates that the distributed storage cluster is under temporary business processing pressure, and the timeout processing process is skipped.

[0097] B5: When the target object storage processes that are not in a slow disk state are concentrated on the target number of nodes, it is determined whether the node state of each node is in a sub-healthy network state.

[0098] When the target object storage processes that are not in the slow disk state are concentrated on the target number of nodes, it is possible that these target number of nodes cause IO timeout problems. Therefore, it is necessary to determine whether each target number of nodes is in the network sub-health state.

[0099] B6: When the node is in a sub-healthy network state, the target object storage process on the node is treated as an abnormal object storage process.

[0100] For example, when a node is in a sub-healthy network state, the abnormal object storage process of the node can be isolated by shutting down the node network port, but the abnormal object storage process on the node still needs to meet the timeout isolation policy.

[0101] B7: When the node is not in the network sub-health state, continue to determine whether the node is in the node sub-health state.

[0102] In the actual implementation process, whether a node is in a sub-healthy state can be judged comprehensively based on node-level related software and hardware health information such as the node's CPU utilization and node memory status.

[0103] B8: When the node status of the node is in a sub-healthy state, the target object storage process on the node is treated as an abnormal object storage process.

[0104] B9: When the node is not in a network sub-health state and the node is not in a node sub-health state, the cluster monitoring process reports an unknown timeout prompt message.

[0105] If the Monitor fails to locate the IO timeout during the IO timeout location determination process, it is considered that the IO timeout is caused by an unknown reason. By reporting an unknown timeout prompt message, manual intervention is prompted to further locate the cause in detail.

[0106] In a feasible implementation manner, the process of executing the timeout isolation policy on the abnormal object storage process includes the following steps:

[0107] C1: For each abnormal object storage process, determine whether the storage pool corresponding to the abnormal object storage process is in a state close to exceeding the fault domain.

[0108] Specifically, for each abnormal object storage process, determine whether the storage pool corresponding to the abnormal object storage process is in a state close to the super fault domain, including: for each abnormal object storage process, traverse the storage group corresponding to the abnormal object storage process, that is, PG; when at least one storage group corresponding to the abnormal object storage process is in a degraded state and at least one data in the storage group does not have redundancy, determine that the storage pool corresponding to the abnormal object storage process is in a state close to the super fault domain.

[0109] Usually, when an OSD process is marked as down, the PG mapped to the OSD process will be converted to a degraded state; if at least one storage group, i.e., PG, corresponding to an abnormal object storage process is already in a degraded state, and at least one data in the storage group, i.e., PG, is not redundant, then if the abnormal object storage process is isolated again, it will cause the storage pool to exceed the fault domain.

[0110] For example, if the storage pool of a distributed storage cluster adopts a three-copy storage method, under normal conditions, a piece of data has three copies of data; if at least one PG corresponding to the abnormal object storage process is already in a degraded state, and at least one data in the PG currently has only one copy of data in the storage pool, the data has no redundancy, and the storage pool is in a state close to the super fault domain.

[0111] For example, if the storage pool of a distributed storage cluster adopts a 4+2 erasure storage method, that is, one piece of data corresponds to 4 data blocks and 2 check blocks and is located on different disks, then 2 disk failures are allowed. If at least one PG corresponding to the abnormal object storage process is already in a degraded state, and at least one data in the PG has only 4 disks left in the storage pool, the data has no redundancy, and the storage pool is in a state close to the super fault domain.

[0112] C2: When the storage pool corresponding to the abnormal object storage process is not in a state close to the super fault domain, isolate the abnormal object storage process.

[0113] C3: When the storage pool corresponding to the abnormal object storage process is close to the fault domain super state, the abnormal object storage process is not isolated.

[0114] During the actual implementation process, an IO timeout detection module is deployed in the OSD process. The IO timeout detection module is used to determine whether the OSD process has an IO timeout problem. The cluster monitoring process includes an IO timeout problem positioning module and an IO timeout fault handling module. The IO timeout problem positioning module is used by Monitor to determine the abnormal object storage process; the IO timeout fault handling module is used to execute the timeout isolation strategy for the abnormal object storage process.

[0115] Reference Figure 2 , shows an execution flow chart of the IO timeout detection module in an embodiment of the present application. The process of the IO timeout detection module determining whether an IO timeout problem occurs in the OSD process and reporting a request timeout message may include the following process:

[0116] First, determine whether the request processing time is greater than or equal to the timeout threshold;

[0117] If the request processing time is less than the timeout threshold, the process ends;

[0118] If the request processing time is greater than or equal to the timeout threshold, continue to determine whether a request timeout message has been reported;

[0119] If it has not been reported, a request timeout message will be reported to the cluster monitoring process;

[0120] If it has been reported, continue to determine whether the time interval from the last report exceeds the target reporting interval;

[0121] If the time interval from the last report does not exceed the target reporting interval, the process ends;

[0122] If the time interval from the last report exceeds the target reporting interval, a request timeout message is reported to the cluster monitoring process.

[0123] Reference Figure 3 , shows an execution flow chart of the IO timeout problem location module in an embodiment of the present application. The process of the IO timeout problem location module for Monitor to determine the abnormal object storage process includes:

[0124] First, determine whether all disks corresponding to the target object storage processes are in slow disk state;

[0125] If all are in slow disk state, each target object storage process will be treated as an abnormal object storage process;

[0126] If not all of them are in slow disk state, determine whether the target object storage processes that are not in slow disk state are concentrated in the target number of nodes;

[0127] If the number of nodes is not concentrated in the target number, it indicates that the cluster is overloaded due to temporary excessive business pressure, and the timeout process is terminated.

[0128] If the number of nodes is concentrated in the target number, continue to determine whether each node is in a sub-healthy state of the network;

[0129] If the node is in sub-healthy network condition, the target object storage process on the node will be treated as an abnormal object storage process;

[0130] If the node is not in the network sub-health state, continue to determine whether the node is in the node sub-health state;

[0131] If the node is in a sub-healthy state, the target object storage process on the node will be treated as an abnormal object storage process;

[0132] If the node is not in a sub-healthy state, an unknown timeout prompt message will be reported.

[0133] Reference Figure 4 , shows an execution flow chart of the IO timeout fault processing module in an embodiment of the present application. The process of the IO timeout fault processing module for executing the timeout isolation strategy for the abnormal object storage process includes:

[0134] For each abnormal object storage process, traverse the storage group corresponding to the abnormal object storage process, that is, PG;

[0135] Determining whether at least one storage group is in a degraded state and whether at least one data in the storage group has no redundancy;

[0136] If at least one storage group is in a degraded state and at least one data does not have redundancy, the abnormal object storage process is not isolated;

[0137] If at least one storage group is not in a degraded state or at least one data has redundancy, the abnormal object storage process is isolated.

[0138] This method monitors IO timeouts in the business processing flow of a distributed storage cluster. When an IO timeout occurs, the request timeout information is reported to the cluster monitoring process, and the abnormal OSD process that causes the slow OP in the cluster is analyzed. Under the premise that the storage pool does not exceed the fault domain, the abnormal OSD process is isolated to ensure that the remaining normal OSD processes are not affected by the abnormal OSD process, so as to support the cluster to provide normal services. By automatically locating the abnormal OSD process and isolating it, labor costs can be reduced, the efficiency of handling IO failures can be improved, and the impact on the business of the distributed storage cluster can be reduced.

[0139] Reference Figure 5 , shows a functional module diagram of a request timeout processing device provided by an embodiment of the present application, which is applied to a distributed storage cluster, and the device includes:

[0140] A timeout reporting module 100 is configured to monitor whether each object storage process in the distributed storage cluster has timed out its own request processing time, and report a request timeout message to a cluster monitoring process of the distributed storage cluster if the timed-out object storage process has timed out. The request timeout message reported by the timed-out object storage process includes the target object storage process that caused the object storage process to time out.

[0141] The timeout isolation module 200 is used by the cluster monitoring process to determine whether the target object storage process is an abnormal object storage process based on the disk status corresponding to the target object storage process and the node status where the target object storage process is located, and to execute a timeout isolation strategy for the abnormal object storage process.

[0142] Reference Figure 6 , shows a schematic diagram of a computer device provided in an embodiment of the present application, including: at least one processor 601, and a memory 602, wherein the memory 602 stores a computer program that can be run on the processor 601, wherein the processor 601 executes the request timeout processing method described in the embodiment when executing the computer program.

[0143] Reference Figure 7 , shows a schematic diagram of a non-volatile readable storage medium provided in an embodiment of the present application, wherein the non-volatile readable storage medium 700 stores a computer program 701, wherein when the computer program 701 is executed by the processor, the request timeout processing method described in the embodiment is executed.

[0144] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0145] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0146] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0147] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0148] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0149] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0150] Finally, it needs to be pointed out that in this document, relational terms such as first and second and the like can only be intended to distinguish one entity or operation from another entity or operation without necessarily requiring or implying any actual such relationship or order between such entities or operations. Moreover, the terms "comprising", "comprising" or any other variations thereof are intended to cover non-exclusive inclusions, so that a process, method, article or terminal device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or terminal device. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or terminal device comprising the element.

[0151] The principles and implementation manners of the present application are described by applying specific examples in this document, and the above example descriptions are only used to help understand the method and its core idea of the present application; meanwhile, for the general technical personnel in the field, the specific implementation manners and application ranges will be changed according to the idea of the present application, and the above description should not be understood as the limitation of the present application.

Claims

1. A method for processing a request timeout, characterized in that: Applied to a distributed storage cluster, the method includes: Each object storage process in the distributed storage cluster monitors whether its own request processing time has timed out, and the object storage process with timeout reports a request timeout message to the cluster monitoring process of the distributed storage cluster, wherein the request timeout message reported by the object storage process with timeout includes the target object storage process that caused the object storage process to timeout; The cluster monitoring process determines whether the target object storage process is an abnormal object storage process according to the disk status corresponding to the target object storage process and the node status where the target object storage process is located, and executes a timeout isolation strategy for the abnormal object storage process.

2. The method according to claim 1, characterized in that Each object storage process in the distributed storage cluster monitors whether its request processing time has timed out, and the object storage process with timeout reports a request timeout message to the cluster monitoring process of the distributed storage cluster, including: Each object storage process in the distributed storage cluster monitors its own request processing time at a predetermined time interval to determine whether it is greater than or equal to a timeout threshold. If the request processing time is greater than or equal to the timeout threshold, a timeout problem occurs. The object storage process with timeout determines whether a request timeout message corresponding to the current request has been reported; When the request timeout message has not been reported, or the time interval since the last reporting of the request timeout message exceeds the target reporting interval, the object storage process reports the request timeout message to the cluster monitoring process of the distributed storage cluster.

3. The method according to claim 2, characterized in that After the object storage process reports the request timeout message to the cluster monitoring process of the distributed storage cluster, the method further includes: The object storage process detects whether the request with the timeout problem has been processed; When the request with the timeout problem is processed, the object storage process reports a cancellation request corresponding to the request timeout message to the cluster monitoring process; In response to the cancellation request, the cluster monitoring process deletes the request timeout message of the object storage process stored in the cluster monitoring process.

4. The method according to claim 1, wherein The cluster monitoring process determines whether the target object storage process is an abnormal object storage process according to the disk status corresponding to the target object storage process and the node status where the target object storage process is located, including: The cluster monitoring process determines whether the disk statuses corresponding to the target object storage processes are all in slow disk status; When all the disks corresponding to the target object storage processes are in slow disk state, all the target object storage processes are regarded as abnormal object storage processes; When all disks corresponding to the target object storage processes are not all in slow disk state, determine whether the target object storage processes that are not in slow disk state are concentrated in the target number of nodes; If the target object storage processes that are not in a slow disk state are not concentrated on the target number of nodes, it indicates that the distributed storage cluster is under temporary business processing pressure, and the timeout processing process is skipped; When the target object storage processes that are not in the slow disk state are concentrated on the target number of nodes, determine whether the node status of each node is in the network sub-health state; When the node is in a sub-healthy network state, the target object storage process on the node is treated as an abnormal object storage process; When the node is not in a network sub-health state and the node state of the node is in a node sub-health state, the target object storage process on the node is regarded as an abnormal object storage process.

5. The method according to claim 4, characterized in that When the target object storage processes that are not in a slow disk state are concentrated on a target number of nodes, and after determining whether each node is in a network sub-health state, the method further includes: When the node is not in a network sub-health state and the node is not in a node sub-health state, the cluster monitoring process reports an unknown timeout prompt message.

6. The method according to claim 1, characterized in that Executing a timeout isolation strategy for the abnormal object storage process includes: For each abnormal object storage process, determine whether the storage pool corresponding to the abnormal object storage process is in a state close to the super fault domain; When the storage pool corresponding to the abnormal object storage process is not in a state close to the super fault domain, isolate the abnormal object storage process; When the storage pool corresponding to the abnormal object storage process is in a state close to exceeding the fault domain, the abnormal object storage process is not isolated.

7. The method according to claim 6, characterized in that For each abnormal object storage process, determine whether the storage pool corresponding to the abnormal object storage process is in a state close to exceeding the fault domain, including: For each abnormal object storage process, traverse the storage group corresponding to the abnormal object storage process; When at least one storage group corresponding to the abnormal object storage process is in a degraded state and at least one data in the storage group has no redundancy, it is determined that the storage pool corresponding to the abnormal object storage process is in a state close to a super fault domain.

8. A device for processing request timeout, characterized in that: Applied to a distributed storage cluster, the device includes: A timeout reporting module is configured to monitor whether each object storage process in the distributed storage cluster has timed out its own request processing time, and the object storage process with timeout reports a request timeout message to the cluster monitoring process of the distributed storage cluster, wherein the request timeout message reported by the object storage process with timeout includes the target object storage process that caused the object storage process to time out; The timeout isolation module is used for the cluster monitoring process to determine whether the target object storage process is an abnormal object storage process based on the disk status corresponding to the target object storage process and the node status where the target object storage process is located, and to execute the timeout isolation strategy for the abnormal object storage process.

9. A computer device, characterized in that: include: At least one processor and a memory, wherein the memory stores a computer program that can be run on the processor, wherein when the processor executes the computer program, the request timeout processing method according to any one of claims 1 to 7 is executed.

10. A non-volatile readable storage medium, characterized in that: The non-volatile readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the method for processing a request timeout according to any one of claims 1 to 7 is executed.

Images