Request timeout processing method and product

Abstract

The embodiment of the invention provides a request timeout processing method and product, and the method comprises the steps: monitoring whether the request processing time of each object storage process in a distributed storage cluster is timeout or not through each object storage process; the overtime object storage process reports a request overtime message to a cluster monitoring process of the distributed storage cluster, wherein the request overtime message reported by the overtime object storage process comprises a target object storage process causing the overtime of the object storage process; the cluster monitoring process determines whether the target object storage process is an abnormal object storage process or not according to the disk state corresponding to the target object storage process and the node state where the target object storage process is located, and executes an overtime isolation strategy on the abnormal object storage process. The embodiment of the invention aims to reduce the service influence on the distributed storage cluster.

Description

Technical Field

[0001] Embodiments of the present application relate to the technical field of data storage, and in particular, to a method and product for processing request timeouts. Background Art

[0002] During the normal processing of business read and write requests in the distributed storage cluster, network fluctuations, node resource exhaustion, and disk failures may cause abnormal phenomena such as "slow disk". These abnormal scenarios will affect some OSD processes in the distributed storage cluster, thereby causing IO processing to be affected accordingly.

[0003] However, since the data security of the distributed storage cluster needs to be protected by redundancy, the storage group PG corresponding to a requested data usually includes multiple OSD processes, and the OSD process members of each PG need to interact. Therefore, if the IO processing of some OSD processes in the distributed storage cluster slows down or even blocks, the processing flow of other OSDs will also be affected, eventually leading to slow requests (slow op) in the distributed storage cluster and even business interruption of the distributed storage cluster.

[0004] Currently, when a slow op occurs in a distributed storage cluster, there is no standard processing flow that can be quickly processed to allow the distributed storage cluster to adaptively recover from the op blocking state. Because the reasons for the occurrence of slow op in the distributed storage cluster are many and complex, the current conventional processing method is manual analysis, and corresponding manual processing is performed according to the detailed cause of the problem after locating to help the distributed storage cluster recover.

[0005] However, when using manual analysis, the process from problem discovery to human intervention and finally solving the problem will result in excessively high labor costs and be time-consuming, causing the business of the distributed storage cluster to be affected for a long time. Summary of the invention

[0006] The embodiments of the present application provide a method and product for processing request timeouts, aiming to reduce the impact on the services of a distributed storage cluster.

[0007] In a first aspect, an embodiment of the present application provides a method for processing a request timeout, which is applied to a distributed storage cluster, and the method includes: Each object storage process in the distributed storage cluster monitors whether its request processing time has timed out, and the object storage process with timeout reports a request timeout message to the cluster monitoring process of the distributed storage cluster, and the request timeout message reported by the object storage process with timeout includes the target object storage process that caused the object storage process to timeout; The cluster monitoring process determines whether the target object storage process is an abnormal object storage process according to the disk status corresponding to the target object storage process and the node status where the target object storage process is located, and executes a timeout isolation strategy for the abnormal object storage process.

[0008] Optionally, each object storage process in the distributed storage cluster monitors whether its request processing time has timed out, and the object storage process with timeout reports a request timeout message to the cluster monitoring process of the distributed storage cluster, including: Each object storage process in the distributed storage cluster monitors whether its own request processing time is greater than or equal to a timeout threshold at a calibrated time interval. When the request processing time is greater than or equal to the timeout threshold, it indicates that a timeout problem exists; The object storage process with timeout determines whether a request timeout message corresponding to the current request has been reported; When the request timeout message has not been reported, or the time interval from the last reporting of the request timeout message exceeds the target reporting interval, the object storage process reports the request timeout message to the cluster monitoring process of the distributed storage cluster.

[0009] Optionally, after the object storage process reports the request timeout message to the cluster monitoring process of the distributed storage cluster, the method further includes: The object storage process detects whether the request with the timeout problem has been processed; When the request with the timeout problem is processed, the object storage process reports a cancellation request corresponding to the request timeout message to the cluster monitoring process; In response to the cancellation request, the cluster monitoring process deletes the request timeout message of the object storage process stored in the cluster monitoring process.

[0010] Optionally, the cluster monitoring process determines whether the target object storage process is an abnormal object storage process according to the disk status corresponding to the target object storage process and the node status where the target object storage process is located, including: The cluster monitoring process determines whether the disk states corresponding to the target object storage processes are all in a slow disk state; When all disks corresponding to all target object storage processes are in slow disk state, all target object storage processes are regarded as abnormal object storage processes; When not all disks corresponding to all target object storage processes are in a slow disk state, it is determined whether the target object storage processes that are not in a slow disk state are concentrated in the target number of nodes; When the target object storage processes that are not in the slow disk state are not concentrated on the target number of nodes, it indicates that the distributed storage cluster is under temporary business processing pressure, and the timeout processing process is skipped; When the target object storage processes that are not in a slow disk state are concentrated on the target number of nodes, it is determined whether the node state of each node is in a network sub-health state; When the node is in a sub-healthy network state, the target object storage process on the node is regarded as an abnormal object storage process; When the node is not in a network sub-health state, and the node state of the node is in a node sub-health state, the target object storage process on the node is regarded as an abnormal object storage process.

[0011] Optionally, when each target object storage process that is not in a slow disk state is concentrated on a target number of nodes, after determining whether each node is in a network sub-health state, the method further includes: When the node is not in a network sub-health state and the node is not in a node sub-health state, the cluster monitoring process reports an unknown timeout prompt message.

[0012] Optionally, executing a timeout isolation strategy for the abnormal object storage process includes: For each abnormal object storage process, determine whether the storage pool corresponding to the abnormal object storage process is in a state close to the super fault domain; When the storage pool corresponding to the abnormal object storage process is not in a state close to the super fault domain, the abnormal object storage process is isolated; When the storage pool corresponding to the abnormal object storage process is in a state close to the super fault domain, the abnormal object storage process is not isolated.

[0013] Optionally, for each abnormal object storage process, determining whether a storage pool corresponding to the abnormal object storage process is in a state close to a super fault domain includes: For each abnormal object storage process, traverse the storage group corresponding to the abnormal object storage process; When at least one storage group corresponding to the abnormal object storage process is in a degraded state and at least one data in the storage group has no redundancy, it is determined that the storage pool corresponding to the abnormal object storage process is in a state close to a super fault domain.

[0014] In a second aspect, an embodiment of the present application provides a request timeout processing device, which is applied to a distributed storage cluster, and the device includes: A timeout reporting module is used for each object storage process in the distributed storage cluster to monitor whether its request processing time has timed out. The object storage process with timeout reports a request timeout message to the cluster monitoring process of the distributed storage cluster. The request timeout message reported by the object storage process with timeout includes the target object storage process that caused the object storage process to time out. The timeout isolation module is used for the cluster monitoring process to determine whether the target object storage process is an abnormal object storage process according to the disk status corresponding to the target object storage process and the node status where the target object storage process is located, and to execute the timeout isolation strategy for the abnormal object storage process.

[0015] In a third aspect, an embodiment of the present application provides a computer device, comprising: at least one processor, and a memory, wherein the memory stores a computer program that can be run on the processor, wherein when the processor executes the computer program, the request timeout processing method described in the first aspect of the embodiment is executed.

[0016] In a fourth aspect, an embodiment of the present application provides a non-volatile readable storage medium, wherein the non-volatile readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the request timeout processing method described in the first aspect of the embodiment is executed.

[0017] Beneficial effects: In the processing method provided in this embodiment, each object storage process in the distributed storage cluster monitors whether its own request processing time has timed out, and the object storage process with timeout reports a request timeout message to the cluster monitoring process of the distributed storage cluster, and the request timeout message reported by the object storage process with timeout includes the target object storage process that caused the object storage process to timeout; the cluster monitoring process determines whether the target object storage process is an abnormal object storage process based on the disk status corresponding to the target object storage process and the node status where the target object storage process is located, and executes a timeout isolation strategy for the abnormal object storage process.

[0018] When an object storage process in a distributed storage device monitors its own request processing time and a timeout occurs, the target object storage process that causes the timeout is reported to the cluster monitoring process through a request timeout message. The cluster monitoring process locates the abnormal object storage process according to the disk status of the target object storage process and the node status of each target object storage process, and determines whether to isolate the abnormal object storage process. In this way, the abnormal object storage process can be automatically isolated when an IO timeout occurs. Compared with manual processing of IO timeout problems, the timeliness of processing can be significantly improved in the process of processing IO timeout problems, and the time that the IO timeout problem affects the business of the distributed storage cluster can be greatly reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following is a brief introduction to the drawings required for use in the embodiments or related technical descriptions.

[0020] Figure 1 A flowchart showing a method for processing a request timeout provided in an embodiment of the present application is shown; Figure 2 The following is a flowchart showing an execution of an IO timeout detection module provided in an embodiment of the present application; Figure 3 The following is a flowchart of an IO timeout problem location module provided in an embodiment of the present application; Figure 4 The following is a flowchart of an IO timeout fault processing module according to an embodiment of the present application; Figure 5 A functional module diagram of a request timeout processing device provided in an embodiment of the present application is shown; Figure 6 A schematic diagram of a computer device provided in an embodiment of the present application is shown; Figure 7 A schematic diagram of a non-volatile readable storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0022] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, each embodiment of the present application will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in each embodiment of the present application, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can also be implemented. The division of the following embodiments is for the convenience of description, and the specific implementation of the present application should not constitute any limitation, and the various embodiments can be combined with each other and referenced to each other without contradiction.

[0023] OSD: Object-based Storage Device, object storage process; Monitor: cluster monitoring service; Slow Op: slow request (i.e. IO timeout request); The Op blocking state refers to the state of a process when it is waiting for an event or resource to be ready. When a process requests the use and allocation of a resource, or waits for an event to occur (such as the completion of an I / O operation), it will switch from the running state to the blocking state. In the blocking state, even if the processor is assigned to the process, it cannot be executed because the process needs to wait for the resources to be ready.

[0024] During the normal processing of business read and write requests in the distributed storage cluster, network fluctuations, node resource exhaustion, and disk failures may cause abnormal phenomena such as "slow disk". These abnormal scenarios will affect some OSD processes in the distributed storage cluster, thereby causing IO processing to be affected accordingly.

[0025] However, since the data security of the distributed storage cluster needs to be protected by redundancy, the storage group PG corresponding to a requested data usually includes multiple OSD processes, and the OSD process members of each PG need to interact. For example, taking the three-copy storage method as an example, for the PG corresponding to one data, there will be three OSD processes that store the data in three hard disks respectively. The three OSD processes include a master OSD process and two slave OSD processes, that is, the OSD processes need to interact with each other.

[0026] Therefore, if the IO processing of some OSD processes in the distributed storage cluster slows down or even gets blocked, the processing flow of other OSDs will also be affected, eventually leading to slow requests in the distributed storage cluster and even business interruption in the distributed storage cluster.

[0027] Currently, when a slow op occurs in a distributed storage cluster, there is no standard processing flow that can be quickly processed to allow the distributed storage cluster to adaptively recover from the op blocking state. Because the reasons for the slow op in the distributed storage cluster are very many and complex, the current conventional processing method is manual analysis, and corresponding manual processing is performed according to the detailed cause of the problem after positioning to help the distributed storage cluster recover to normal; however, in the manual analysis method, the process from problem discovery to manual intervention to the final solution will result in excessively high labor costs and a long time, resulting in the business of the distributed storage cluster being affected for a long time.

[0028] Therefore, in order to reduce the impact on the services of the distributed storage cluster, an embodiment of the present application provides a method for processing a request timeout.

[0029] Reference Figure 1 , shows a flowchart of a method for processing a request timeout provided by an embodiment of the present application, the method is applied to a distributed storage cluster and may include the following steps: S101: Each object storage process in the distributed storage cluster monitors whether its own request processing time has timed out. The object storage process with timeout reports a request timeout message to the cluster monitoring process of the distributed storage cluster. The request timeout message reported by the object storage process with timeout includes the target object storage process that caused the object storage process to timeout.

[0030] In a distributed storage cluster, multiple object storage processes are maintained, that is, multiple OSD processes are maintained. Usually, one OSD process corresponds to one disk. Due to resource limitations such as disk hardware failure, high CPU usage or high memory usage, if the IO processing of some OSD processes slows down or even blocks, the processing flow of other OSDs will also be affected.

[0031] Therefore, in this embodiment, each object storage process in the distributed storage cluster monitors its own request processing time and when a timeout problem occurs, the object storage process monitors its own request processing time and reports the request timeout message to the cluster monitoring process of the distributed storage cluster, that is, reports to the Monitor of the distributed storage cluster.

[0032] In a feasible implementation manner, each object storage process in the distributed storage cluster monitors whether its request processing time has timed out, and the process in which the object storage process with timeout reports a request timeout message to the cluster monitoring process of the distributed storage cluster is as follows: A1: Each object storage process in the distributed storage cluster monitors whether its own request processing time is greater than or equal to a timeout threshold at a calibrated time interval.

[0033] For example, in order to avoid the OSD process detecting IO timeouts in real time and reporting them, which results in excessive processing costs, a timer can be set for the OSD process. The duration of the timer is configured as a calibration time interval. For example, the calibration time interval can be set to 10s, that is, the OSD process detects whether it has an IO timeout problem every 10s.

[0034] When the OSD process handles the current IO request, it records the request processing time, compares it with the preset timeout threshold, and determines whether the OSD process itself has an IO timeout problem; the timeout threshold can be set according to the actual application requirements of the distributed storage cluster, and this embodiment does not impose any restrictions.

[0035] A2: When the request processing time is greater than or equal to the timeout threshold, it indicates that a timeout problem exists.

[0036] When the object storage process monitors that its own request processing time is less than the timeout threshold, it is necessary to determine whether the current request has not been processed after 10 seconds. If not, it is necessary to determine whether it has timed out again based on the request processing time.

[0037] A3: The object storage process with timeout determines whether a request timeout message corresponding to the current request has been reported.

[0038] Since the cluster monitoring process takes time to locate IO timeouts, if an IO timeout occurs in the OSD process, it can determine whether a request timeout message corresponding to the current request has been reported, and the time interval between the last report and the current time to determine whether to continue reporting the request timeout message corresponding to the current request.

[0039] The request timeout message reported by the OSD process includes the target OSD process that caused the OSD process to time out. For example, if in the three-copy storage method of data A, the processing process of the master OSD process 1 causes the IO timeout of the slave OSD process 2 and the slave OSD process 3, and then the request timeout messages reported by the slave OSD process 2 and the slave OSD process 3 include the master OSD process 1.

[0040] When the OSD process has not reported a request timeout message corresponding to the current request, the request timeout message can be reported to the cluster monitoring process, i.e., Monitor.

[0041] A4: When the request timeout message has not been reported, or the time interval from the last reporting of the request timeout message exceeds the target reporting interval, the object storage process reports the request timeout message to the cluster monitoring process of the distributed storage cluster.

[0042] If the OSD process has reported a request timeout message corresponding to the current request, but the time interval since the last reporting of the request timeout message has exceeded the target reporting interval, the request timeout message can be reported to the Monitor again.

[0043] If the OSD process has reported a request timeout message corresponding to the current request, but the time interval since the last reporting of the request timeout message is still less than the target reporting interval, the request timeout message will no longer be reported to the Monitor repeatedly.

[0044] Specifically, after the object storage process reports the request timeout message to the cluster monitoring process of the distributed storage cluster, the object storage process may also detect whether the request with the timeout problem has been processed.

[0045] When the request with the timeout problem is processed, the object storage process reports a cancellation request corresponding to the request timeout message to the cluster monitoring process; the cluster monitoring process responds to the cancellation request and deletes the request timeout message of the object storage process stored in itself.

[0046] That is, if the reported OSD process currently has no timeout request, a request timeout message to cancel the OSD process is reported to the Monitor, and the Monitor deletes the request timeout message of the OSD process from the data structure that has been reported in its own record.

[0047] S102: The cluster monitoring process determines whether the target object storage process is an abnormal object storage process according to the disk status corresponding to the target object storage process and the node status where the target object storage process is located, and executes a timeout isolation strategy for the abnormal object storage process.

[0048] In a feasible implementation manner, the cluster monitoring process determines whether the target object storage process is an abnormal object storage process according to the disk status corresponding to the target object storage process and the node status where the target object storage process is located, and the process may include: B1: The cluster monitoring process determines whether the disk states corresponding to the target object storage processes are all in a slow disk state.

[0049] Specifically, in this embodiment, the disk state includes a slow disk state or a non-slow disk state, wherein the slow disk state refers to a phenomenon that the disk hardware corresponding to the OSD process is damaged or faulty.

[0050] In the actual implementation process, the cluster monitoring process monitors and obtains the disk status of the target object storage process.

[0051] At the current moment, the number of target OSD processes received by the cluster monitoring process may be only one or more.

[0052] For example, OSD1, OSD2, and OSDn all report OSD0 as the target object storage process that caused their own IO timeouts, and OSD3 and OSD4 report OSD5 as the target object storage process that caused their own IO timeouts. The Monitor first determines whether the disk status of OSD0 and OSD5 are all in a slow disk state, that is, whether the IO processing timeout is caused by the disk failure corresponding to OSD0 and OSD5.

[0053] For example, OSD1-OSDn all report OSD0 as the target object storage process that caused their own IO timeout. At this time, Monitor only receives OSD0 as the target OSD process, and then determines whether the disk status corresponding to OSD0 is a slow disk status.

[0054] In the actual implementation process, it is also possible to detect whether the disk status corresponding to the OSD process with timeout is a slow disk status. For example, if OSD1, OSD2 and OSDn all report OSD0 as the target object storage process that caused their own IO timeout, in addition to judging whether OSD0 is in a slow disk status, it is also possible to judge whether OSD1, OSD2 and OSDn themselves are in a slow disk status, that is, to determine whether the IO timeouts of OSD1, OSD2 and OSDn are due to their own problems rather than OSD0; if the OSD process that reports other OSD processes as causing its own IO timeout is actually due to its own abnormal state of slow disk, the OSD process can be regarded as an abnormal object storage process, and then determine whether to isolate and judge the OSD process to avoid the OSD process from affecting other OSD processes in the future.

[0055] B2: When all disks corresponding to all target object storage processes are in slow disk state, all target object storage processes are regarded as abnormal object storage processes.

[0056] For example, when OSD0 and OSD5 are all OSD0 and OSD5, OSD0 and OSD5 are both stored as abnormal object processes.

[0057] B3: When not all disks corresponding to all target object storage processes are in a slow disk state, it is determined whether the target object storage processes that are not in a slow disk state are concentrated in a target number of nodes.

[0058] Usually, the number of OSDs on each node of a distributed storage cluster and the number of OSDs and PGs are balanced. If the cluster is overloaded due to temporary excessive business pressure, causing the distributed storage cluster to generate slow ops, the OSDs that have timed out through IO will be distributed on each node, and the number distribution is relatively uniform. Therefore, based on the distribution of OSDs that have timed out through IO, it can be determined whether the IO timeout is caused by problems with certain nodes.

[0059] For example, the target number can be 1, that is, whether the storage processes of each target object that is not in a slow disk state are concentrated in one node. The target number can also be set according to the needs of the actual application, but the configuration of the target number usually does not exceed the data redundancy number of the distributed storage cluster. For example, when the distributed storage cluster adopts a 3-copy method, the configuration of the target number does not exceed 3.

[0060] B4: When the target object storage processes that are not in the slow disk state are not concentrated on the target number of nodes, it indicates that the distributed storage cluster is in a temporary business processing pressure state, and the timeout processing process is skipped.

[0061] B5: When the target object storage processes that are not in a slow disk state are concentrated on the target number of nodes, it is determined whether the node state of each node is in a network sub-health state.

[0062] When the target object storage processes that are not in the slow disk state are concentrated on the target number of nodes, it is possible that these target number of nodes cause the IO timeout problem. Therefore, it is necessary to determine whether each target number of nodes is in the network sub-health state.

[0063] B6: When the node is in a sub-healthy network state, the target object storage process on the node is treated as an abnormal object storage process.

[0064] For example, when a node is in a sub-healthy network state, the abnormal object storage process of the node can be isolated by shutting down the node network port, but the abnormal object storage process on the node still needs to meet the timeout isolation policy.

[0065] B7: When the node is not in the network sub-health state, continue to determine whether the node is in the node sub-health state.

[0066] In the actual implementation process, whether a node is in a sub-healthy state can be judged comprehensively based on the node-level related software and hardware health information such as the node's CPU utilization and node memory status.

[0067] B8: When the node status of the node is in a sub-healthy state, the target object storage process on the node is regarded as an abnormal object storage process.

[0068] B9: When the node is not in a network sub-health state and the node is not in a node sub-health state, the cluster monitoring process reports an unknown timeout prompt message.

[0069] If the Monitor fails to locate the IO timeout during the IO timeout location determination process, it is considered that the IO timeout is caused by an unknown reason. By reporting an unknown timeout prompt message, manual intervention is prompted to further locate the cause in detail.

[0070] In a feasible implementation manner, the process of executing the timeout isolation strategy on the abnormal object storage process includes the following steps: C1: For each abnormal object storage process, determine whether the storage pool corresponding to the abnormal object storage process is in a state close to the super fault domain.

[0071] Specifically, for each abnormal object storage process, determine whether the storage pool corresponding to the abnormal object storage process is in a state close to the super fault domain, including: for each abnormal object storage process, traverse the storage group corresponding to the abnormal object storage process, that is, PG; when at least one storage group corresponding to the abnormal object storage process is in a degraded state and at least one data in the storage group does not have redundancy, determine that the storage pool corresponding to the abnormal object storage process is in a state close to the super fault domain.

[0072] Usually, when an OSD process is marked as down, the PG mapped to the OSD process will be converted to a degraded state; if at least one storage group, i.e., PG, corresponding to an abnormal object storage process is already in a degraded state, and at least one data in the storage group, i.e., PG, is not redundant, then if the abnormal object storage process is isolated again, it will cause the storage pool to exceed the fault domain.

[0073] For example, if the storage pool of a distributed storage cluster adopts a three-copy storage method, under normal conditions, one data has three copies of data; if at least one PG corresponding to the abnormal object storage process is already in a degraded state, and at least one data in the PG currently has only one copy of data in the storage pool, the data has no redundancy, and the storage pool is in a state close to a super fault domain.

[0074] For example, if the storage pool of a distributed storage cluster adopts a 4+2 erasure storage method, that is, one data corresponds to 4 data blocks and 2 check blocks and are located in different disks. At this time, 2 disk failures are allowed. If at least one PG corresponding to the abnormal object storage process is already in a degraded state, and at least one data in the PG has only 4 disks left in the storage pool, the data has no redundancy. At this time, the storage pool is in a state close to the super fault domain.

[0075] C2: When the storage pool corresponding to the abnormal object storage process is not in a state close to the super fault domain, isolate the abnormal object storage process.

[0076] C3: When the storage pool corresponding to the abnormal object storage process is in a state close to the super fault domain, the abnormal object storage process is not isolated.

[0077] In the actual implementation process, an IO timeout detection module is deployed in the OSD process. The IO timeout detection module is used to determine whether an IO timeout problem occurs in the OSD process. The cluster monitoring process includes an IO timeout problem locating module and an IO timeout fault handling module. The IO timeout problem locating module is used by Monitor to determine the abnormal object storage process; the IO timeout fault handling module is used to execute the timeout isolation strategy for the abnormal object storage process.

[0078] Reference Figure 2 , shows an execution flow chart of the IO timeout detection module in an embodiment of the present application. The process of the IO timeout detection module determining whether an IO timeout problem occurs in the OSD process and reporting a request timeout message may include the following process: First, determine whether the request processing time is greater than or equal to the timeout threshold; If the request processing time is less than the timeout threshold, the process ends; If the request processing time is greater than or equal to the timeout threshold, continue to determine whether a request timeout message has been reported; If it has not been reported, a request timeout message is reported to the cluster monitoring process; If it has been reported, continue to determine whether the time interval from the last report exceeds the target reporting interval; If the time interval from the last report does not exceed the target reporting interval, then the process ends; If the time interval from the last report exceeds the target reporting interval, a request timeout message is reported to the cluster monitoring process.

[0079] Reference Figure 3 , shows an execution flow chart of the IO timeout problem location module in an embodiment of the present application. The process in which the IO timeout problem location module is used by Monitor to determine the abnormal object storage process includes: First, determine whether all disks corresponding to the storage processes of each target object are in slow disk state; If all are in slow disk state, each target object storage process is treated as an abnormal object storage process; If not all of them are in the slow disk state, determine whether the target object storage processes that are not in the slow disk state are concentrated in the target number of nodes; If the number of nodes is not the target number, it means that the cluster is overloaded due to temporary excessive business pressure, and the timeout process is terminated. If the nodes are concentrated in the target number, continue to determine whether each node is in a sub-healthy state of the network; If the node is in sub-healthy network, the target object storage process on the node is treated as an abnormal object storage process; If the node is not in a network sub-health state, continue to determine whether the node is in a node sub-health state; If the node is in a sub-healthy state, the target object storage process on the node is treated as an abnormal object storage process; If the node is not in a sub-healthy state, an unknown timeout prompt message is reported.

[0080] Reference Figure 4 , shows an execution flow chart of the IO timeout fault processing module in an embodiment of the present application, and the process of the IO timeout fault processing module for executing the timeout isolation strategy for the abnormal object storage process includes: For each abnormal object storage process, traverse the storage group corresponding to the abnormal object storage process, that is, PG; Determining whether at least one storage group is in a degraded state and whether at least one data in the storage group has no redundancy; If at least one storage group is in a degraded state and at least one data does not have redundancy, the abnormal object storage process is not isolated; If at least one storage group is not in a degraded state or at least one data has redundancy, the abnormal object storage process is isolated.

[0081] This method monitors IO timeouts in the business processing flow of a distributed storage cluster. When an IO timeout occurs, the request timeout information is reported to the cluster monitoring process, and the abnormal OSD process that causes the slow op in the cluster is analyzed. Under the premise that the storage pool does not exceed the fault domain, the abnormal OSD process is isolated to ensure that the remaining normal OSD processes are not affected by the abnormal OSD process, so as to support the cluster to provide normal services. By automatically locating the abnormal OSD process and isolating it, the labor cost can be reduced, the processing efficiency of IO failures can be improved, and the impact on the business of the distributed storage cluster can be reduced.

[0082] Reference Figure 5 , shows a functional module diagram of a request timeout processing device provided in an embodiment of the present application, which is applied to a distributed storage cluster, and the device includes: A timeout reporting module 100 is used for each object storage process in the distributed storage cluster to monitor whether its request processing time has timed out. The object storage process with timeout reports a request timeout message to the cluster monitoring process of the distributed storage cluster. The request timeout message reported by the object storage process with timeout includes the target object storage process that caused the object storage process to time out. The timeout isolation module 200 is used for the cluster monitoring process to determine whether the target object storage process is an abnormal object storage process according to the disk status corresponding to the target object storage process and the node status where the target object storage process is located, and to execute the timeout isolation strategy for the abnormal object storage process.

[0083] Reference Figure 6 , shows a schematic diagram of a computer device provided in an embodiment of the present application, including: at least one processor 601, and a memory 602, wherein the memory 602 stores a computer program that can be executed on the processor 601, wherein the processor 601 executes the request timeout processing method described in the embodiment when executing the computer program.

[0084] Reference Figure 7 , shows a schematic diagram of a non-volatile readable storage medium provided in an embodiment of the present application, wherein the non-volatile readable storage medium 700 stores a computer program 701, wherein the computer program 701, when executed by a processor, executes the request timeout processing method described in the embodiment.

[0085] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0086] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0087] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0088] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0089] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0090] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0091] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0092] Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for processing a request timeout, characterized in that: Applied to a distributed storage cluster, the method includes: Each object storage process in the distributed storage cluster monitors whether its request processing time has timed out, and the object storage process with timeout reports a request timeout message to the cluster monitoring process of the distributed storage cluster, and the request timeout message reported by the object storage process with timeout includes the target object storage process that caused the object storage process to timeout; The cluster monitoring process determines whether the target object storage process is an abnormal object storage process according to the disk status corresponding to the target object storage process and the node status where the target object storage process is located, and executes a timeout isolation strategy for the abnormal object storage process.

2. The method according to claim 1, characterized in that Each object storage process in the distributed storage cluster monitors whether its request processing time has timed out, and the object storage process with timeout reports a request timeout message to the cluster monitoring process of the distributed storage cluster, including: Each object storage process in the distributed storage cluster monitors whether its own request processing time is greater than or equal to a timeout threshold at a calibrated time interval. When the request processing time is greater than or equal to the timeout threshold, it indicates that a timeout problem exists; The object storage process with timeout determines whether a request timeout message corresponding to the current request has been reported; When the request timeout message has not been reported, or the time interval from the last reporting of the request timeout message exceeds the target reporting interval, the object storage process reports the request timeout message to the cluster monitoring process of the distributed storage cluster.

3. The method according to claim 2, characterized in that After the object storage process reports the request timeout message to the cluster monitoring process of the distributed storage cluster, the method further includes: The object storage process detects whether the request with the timeout problem has been processed; When the request with the timeout problem is processed, the object storage process reports a cancellation request corresponding to the request timeout message to the cluster monitoring process; In response to the cancellation request, the cluster monitoring process deletes the request timeout message of the object storage process stored in the cluster monitoring process.

4. The method according to claim 1, characterized in that: The cluster monitoring process determines whether the target object storage process is an abnormal object storage process according to the disk status corresponding to the target object storage process and the node status where the target object storage process is located, including: The cluster monitoring process determines whether the disk states corresponding to the target object storage processes are all in a slow disk state; When all disks corresponding to all target object storage processes are in slow disk state, all target object storage processes are regarded as abnormal object storage processes; When not all disks corresponding to all target object storage processes are in a slow disk state, it is determined whether the target object storage processes that are not in a slow disk state are concentrated in the target number of nodes; When the target object storage processes that are not in the slow disk state are not concentrated on the target number of nodes, it indicates that the distributed storage cluster is under temporary business processing pressure, and the timeout processing process is skipped; When the target object storage processes that are not in a slow disk state are concentrated on the target number of nodes, it is determined whether the node state of each node is in a network sub-health state; When the node is in a sub-healthy network state, the target object storage process on the node is regarded as an abnormal object storage process; When the node is not in a network sub-health state, and the node state of the node is in a node sub-health state, the target object storage process on the node is regarded as an abnormal object storage process.

5. The method according to claim 4, characterized in that When each target object storage process that is not in a slow disk state is concentrated on a target number of nodes, after determining whether each node is in a network sub-health state, the method further includes: When the node is not in a network sub-health state and the node is not in a node sub-health state, the cluster monitoring process reports an unknown timeout prompt message.

6. The method according to claim 1, characterized in that Executing a timeout isolation strategy on the abnormal object storage process includes: For each abnormal object storage process, determine whether the storage pool corresponding to the abnormal object storage process is in a state close to the super fault domain; When the storage pool corresponding to the abnormal object storage process is not in a state close to the super fault domain, the abnormal object storage process is isolated; When the storage pool corresponding to the abnormal object storage process is in a state close to the super fault domain, the abnormal object storage process is not isolated.

7. The method according to claim 6, characterized in that For each abnormal object storage process, determining whether the storage pool corresponding to the abnormal object storage process is in a state close to the super fault domain includes: For each abnormal object storage process, traverse the storage group corresponding to the abnormal object storage process; When at least one storage group corresponding to the abnormal object storage process is in a degraded state and at least one data in the storage group has no redundancy, it is determined that the storage pool corresponding to the abnormal object storage process is in a state close to a super fault domain.

8. A request timeout processing device, characterized in that: Applied to a distributed storage cluster, the device comprises: A timeout reporting module is used for each object storage process in the distributed storage cluster to monitor whether its request processing time has timed out. The object storage process with timeout reports a request timeout message to the cluster monitoring process of the distributed storage cluster. The request timeout message reported by the object storage process with timeout includes the target object storage process that caused the object storage process to time out. The timeout isolation module is used for the cluster monitoring process to determine whether the target object storage process is an abnormal object storage process according to the disk status corresponding to the target object storage process and the node status where the target object storage process is located, and to execute the timeout isolation strategy for the abnormal object storage process.

9. A computer device, characterized in that: include: At least one processor, and a memory, wherein the memory stores a computer program that can be run on the processor, wherein the processor executes the request timeout processing method described in any one of claims 1-7 when executing the computer program.

10. A non-volatile readable storage medium, characterized in that: The non-volatile readable storage medium stores a computer program, wherein the computer program, when executed by a processor, performs the request timeout processing method according to any one of claims 1 to 7.

Images