IO fault location method, device, equipment and computer readable storage medium

Abstract

The embodiments of the present invention relate to the field of distributed storage technology, and disclose an IO fault location method, device, equipment, and computer-readable storage medium. The method includes: obtaining the IO delay of the front-end computing node and the IO delay threshold of the external disk; if the difference between the IO delay of the computing node and the IO delay threshold of the external disk is within a preset numerical range, obtaining the IO delay of the logical unit number and the IO delay threshold of the logical unit number; if the IO delay of the logical unit number is not greater than the IO delay threshold, and the difference between the IO delay of the computing node and the IO delay of the logical unit number is within a preset numerical range, the IO fault is located at the front end; if the difference between the IO delay of the logical unit number and the IO delay threshold is within a preset numerical range, and the difference between the IO delay of the computing node and the IO delay of the logical unit number is within a preset numerical range, the IO fault is located at the back end. In the above manner, the embodiments of the present invention realize the rapid and accurate location of IO faults in distributed storage.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of distributed storage technology, and specifically to an IO fault locating method, device, equipment, and computer-readable storage medium. Background Art

[0002] Distributed storage is currently a popular storage method. It connects multiple storage servers through a network to flexibly allocate storage space, which can improve storage efficiency and storage capacity.

[0003] A distributed storage system generally includes a front-end computing node and a back-end storage node. Various applications can be run on the front-end computing node, and the back-end storage node can save the data of the front-end computing node. During the operation of the distributed storage system, IO failures often occur. In order to enable the distributed storage system to operate normally, it is necessary to locate the IO failure to solve the IO failure. In the process of implementing the embodiments of the present invention, the inventors found that the locating of IO failures in the related art often depends on the alarm information provided by the distributed storage system and the experience of the technicians. The means for locating IO failures are relatively simple, and the efficiency of the locating process is low. Summary of the invention

[0004] In view of the above problems, embodiments of the present invention provide an IO fault location method, apparatus, device and computer-readable storage medium to solve the problem of low efficiency of the distributed storage IO fault location process in the prior art.

[0005] According to one aspect of an embodiment of the present invention, a method for locating an IO fault is provided, wherein the IO fault is an IO fault of distributed storage, and the method includes:

[0006] Obtaining the IO latency of the front-end computing node and the IO latency threshold of the external disk of the computing node;

[0007] If the difference between the IO latency of the computing node and the IO latency threshold of the external disk is within a first preset value range, further obtaining the IO latency of the logical unit number corresponding to the computing node and the IO latency threshold of the logical unit number;

[0008] If the IO latency of the logical unit number is not greater than the IO latency threshold of the logical unit number, and the difference between the IO latency of the computing node and the IO latency of the logical unit number is within a second preset value range, the IO fault is located at the front end;

[0009] If the difference between the IO latency of the logical unit number and the IO latency threshold of the logical unit number is within a third preset value range, and the difference between the IO latency of the computing node and the IO latency of the logical unit number is within a fourth preset value range, the IO fault is located at the back end.

[0010] In an optional manner, after locating the IO fault at the front end, the method further includes:

[0011] If the computing node is a virtual machine, the IO latency of other computing nodes corresponding to the same logical unit number as the computing node, the IO latency of other virtual machines corresponding to the same physical machine as the computing node, and the IO latency of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located are respectively obtained;

[0012] If it is determined that the IO delay of other computing nodes corresponding to the same logical unit number as the computing node is abnormal, the IO fault is located in the interconnection network between the front end and the back end;

[0013] If it is determined that the IO latency of other virtual machines corresponding to the same physical machine as the computing node is normal, and it is determined that the IO latency of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located is normal, the IO fault is located at the computing node;

[0014] If it is determined that the IO latency of other virtual machines corresponding to the same physical machine as the computing node is abnormal, and if it is determined that the IO latency of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located is normal, the IO fault is located on the physical machine where the computing node is located.

[0015] In an optional manner, after locating the IO fault at the front end, the method further includes:

[0016] If the computing node is a physical machine, obtaining the IO latency of other computing nodes corresponding to the same logical unit number as the computing node;

[0017] If it is determined that the IO delay of other computing nodes corresponding to the same logical unit number as the computing node is abnormal, the IO fault is located in the interconnection network between the front end and the back end;

[0018] If it is determined that the IO delays of the other computing nodes corresponding to the same logical unit number as the computing node are normal, the IO fault is located at the computing node.

[0019] In an optional manner, after locating the IO fault at the back end, the method further includes:

[0020] Get the ping network delay of each backend storage node, the average ping network delay of all backend storage nodes, and the ping network delay threshold of the backend storage nodes;

[0021] If the difference between the average value of the ping network delay of all the back-end storage nodes and the ping network delay threshold of the back-end storage node is within the fifth preset value range, and the difference between the ping network delay of each back-end storage node and the ping network delay threshold of the back-end storage node is within the sixth preset value range, the IO fault is located in the Internet network between the back-end storage nodes;

[0022] If the difference between the average value of the ping network delay of all the back-end storage nodes and the ping network delay threshold of the back-end storage nodes is within the seventh preset value range, and the difference between the ping network delay of a preset number of back-end storage nodes and the ping network delay threshold of the back-end storage nodes is within the eighth preset value range, and the ping network delay of other back-end storage nodes other than the preset number of back-end storage nodes is not greater than the ping network delay threshold of the back-end storage nodes, then the IO fault is located in the network of the preset number of back-end storage nodes;

[0023] If the difference between the average value of the ping network delay of all the back-end storage nodes and the ping network delay threshold of the back-end storage nodes is within a ninth preset numerical range, the IO delay of the built-in disks of all the back-end storage nodes and the IO delay threshold of the built-in disks corresponding to the IO delay of the built-in disks are obtained, and the built-in disks whose difference between the IO delay of the built-in disk and the IO delay threshold of the built-in disk is within a tenth preset numerical range are identified as abnormal built-in disks, and the IO fault is located at the abnormal built-in disk.

[0024] According to another aspect of an embodiment of the present invention, an IO fault locating device is provided, wherein the IO fault is an IO fault of distributed storage, and the device comprises:

[0025] A first acquisition module is used to acquire the IO latency of the front-end computing node and the IO latency threshold of the external disk of the computing node;

[0026] A second acquisition module is used to acquire the IO latency of the logical unit number corresponding to the computing node and the IO latency threshold of the logical unit number if the difference between the IO latency of the computing node and the IO latency threshold of the external disk is within a first preset value range;

[0027] A positioning module, configured to locate the IO fault at the front end if the IO delay of the logical unit number is not greater than the IO delay threshold of the logical unit number, and the difference between the IO delay of the computing node and the IO delay of the logical unit number is within a second preset numerical range; and to locate the IO fault at the back end if the difference between the IO delay of the logical unit number and the IO delay threshold of the logical unit number is within a third preset numerical range, and the difference between the IO delay of the computing node and the IO delay of the logical unit number is within a fourth preset numerical range.

[0028] According to another aspect of an embodiment of the present invention, there is provided a computing device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0029] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute the operation of the above-mentioned IO fault locating method.

[0030] According to another aspect of an embodiment of the present invention, an IO fault locating device is provided, wherein the IO fault is an IO fault of distributed storage, and the device includes:

[0031] Collection module, used to collect IO delay information;

[0032] Configuration module, used to configure IO latency threshold;

[0033] A positioning module, used to execute the above-mentioned IO fault positioning method to locate the IO fault;

[0034] The display module is used to obtain the result of the positioning module on the IO fault and display the result.

[0035] In an optional manner, the display module is further used for:

[0036] The IO latency information collected by the collection module is displayed, and the corresponding relationship between the external disk of the front-end computing node and the logical unit number of the back-end storage cluster is displayed.

[0037] In an optional manner, the configuration module is further used to:

[0038] Acquire the configuration information of the front-end computing node and the configuration information of the back-end storage cluster, and send the configuration information of the front-end computing node and the configuration information of the back-end storage cluster to the positioning module;

[0039] The positioning module is also used to locate the IO fault according to the configuration information of the front-end computing node and the configuration information of the back-end storage cluster.

[0040] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein at least one executable instruction is stored in the storage medium. When the executable instruction is executed on a computing device, the computing device executes the operation of the above-mentioned IO fault locating method.

[0041] The embodiment of the present invention obtains the IO delay of the front-end computing node and the IO delay threshold of the external disk of the computing node; if the difference between the IO delay of the computing node and the IO delay threshold of the external disk is within the first preset value range, the IO delay of the logical unit number corresponding to the computing node and the IO delay threshold of the logical unit number are further obtained; if the IO delay of the logical unit number is not greater than the IO delay threshold of the logical unit number, and the difference between the IO delay of the computing node and the IO delay of the logical unit number is within the second preset value range, the IO fault is located at the front end; if the difference between the IO delay of the logical unit number and the IO delay threshold of the logical unit number is within the third preset value range, and the difference between the IO delay of the computing node and the IO delay of the logical unit number is within the fourth preset value range, the IO fault is located at the back end. It can be seen that the embodiment of the present invention can quickly and accurately locate the IO fault by comparing the IO delay of the front-end computing node with the IO delay of the back-end logical unit number.

[0042] The above description is only an overview of the technical solution of the embodiment of the present invention. In order to more clearly understand the technical means of the embodiment of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present invention. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings. In the accompanying drawings:

[0044] Figure 1 A schematic diagram of the process of an IO fault locating method provided by an embodiment of the present invention is shown;

[0045] Figure 2 A schematic diagram showing the structure of an IO fault locating device provided by an embodiment of the present invention is shown;

[0046] Figure 3 A schematic diagram showing the structure of a computing device provided by an embodiment of the present invention is shown;

[0047] Figure 4 A schematic diagram of the structure of an IO fault locating device provided in an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0048] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0049] Figure 1 The flowchart of the embodiment of the IO fault location method of the present invention is shown, and the method is executed by a computing device. In the embodiment of the present invention, the storage space of the computing device stores executable instructions, and the executable instructions enable the processor to execute the IO fault location method. Figure 1 As shown, the method can locate the IO failure of distributed storage, including the following steps:

[0050] Step 110: Obtain the IO latency of the front-end computing node and the IO latency threshold of the external disk of the computing node.

[0051] Among them, the front-end computing node can be various types of PC servers, for example, the front-end computing node can be an x86 server. The external disk of the computing node is located in the storage node on the back-end storage cluster. The back-end storage cluster generally includes multiple storage nodes. Each storage node can include at least one local storage disk. The storage disk on the storage node can be used as an external disk of the front-end computing node. The external disk of the computing node will generate IO delay when executing IO tasks. An IO delay threshold can be set for the external disk of the computing node. The IO delay of the external disk of the computing node is compared with the IO delay threshold of the external disk of the computing node to determine whether the external disk of the computing node has timed out during the execution of the IO task.

[0052] In one implementation of an embodiment of the present invention, the read and write delay wait value of the external disk of the computing node can be obtained through relevant operating system commands, and the obtained wait value can be used as the IO delay of the computing node. The relevant operating system command can be, for example, the iostat command. Further, multiple read and write delay wait values ​​of the external disk of the computing node can be obtained at fixed time intervals, and the moving average of the wait value can be obtained according to the moving average method, and the moving average of the wait value can be used as the IO delay of the computing node. The moving average method is a commonly used method for processing data, which can effectively eliminate abnormal sample data, make the obtained moving average of the wait value more accurate, and make the final IO delay of the computing node more accurate.

[0053] Among them, the distributed storage includes a front-end and a back-end, the front-end includes a plurality of front-end computing nodes, the back-end includes a plurality of back-end storage nodes, and the front-end and the back-end communicate through the Internet. Applications can be run on the front-end computing nodes, and the storage disks on the back-end storage nodes can save the application data of the applications on the front-end computing nodes. The back-end storage nodes can communicate with each other through the Internet between the back-end storage nodes, and each back-end storage node accesses the Internet between the back-end storage nodes through a local network access device. The local network access device of the back-end storage node may include, for example, a network cable and a network card. The IO latency of the computing node may generally include the latency from the computing node to the front-end and back-end Internet, the latency from the front-end and back-end Internet to the storage node, and the internal latency of the storage node. The internal latency of the storage node may also generally include the latency of IO data synchronization between the storage nodes.

[0054] Step 120: If the difference between the IO latency of the computing node and the IO latency threshold of the external disk is within a first preset value range, further obtaining the IO latency of the logical unit number corresponding to the computing node and the IO latency threshold of the logical unit number.

[0055] Among them, the IO latency of the logical unit number can be calculated by the moving average method. The difference between the IO latency of the computing node and the IO latency threshold of the external disk can be calculated first, and then it is determined whether the difference is within the first preset numerical range. If the difference is determined to be within the first preset numerical range, it is determined that the IO latency of the computing node is abnormal, and an IO failure occurs in the distributed storage system. The first preset numerical range can be set according to actual needs. For example, the first preset numerical range can be set to a numerical range of more than 10 times the IO latency threshold of the external disk, that is, when the IO latency of the computing node is greater than 11 times the IO latency threshold of the external disk, it is determined that the difference between the IO latency of the computing node and the IO latency threshold of the external disk is within the first preset numerical range.

[0056] Among them, the logical unit number (LUN, i.e., Logical Unit Number) is a storage unit allocated by the back-end storage cluster to the front-end computing node. The back-end storage cluster generally includes multiple logical unit numbers. The logical unit number of the back-end storage cluster will generate IO delay when executing IO tasks. An IO delay threshold can be set for the logical unit number of the back-end storage cluster, and the IO delay of the logical unit number is compared with the IO delay threshold of the logical unit number to determine whether the logical unit number times out during the execution of the IO task.

[0057] Step 130: If the IO latency of the logical unit number is not greater than the IO latency threshold of the logical unit number, and the difference between the IO latency of the computing node and the IO latency of the logical unit number is within a second preset value range, the IO fault is located at the front end.

[0058] If the IO latency of the logical unit number is not greater than the IO latency threshold of the logical unit number, the IO latency of the logical unit number is determined to be normal. If the difference between the IO latency of the computing node and the IO latency of the logical unit number is further determined to be within a second preset value range, it indicates that an IO fault occurs at the front end, and the IO fault is located at the front end.

[0059] Among them, the second preset numerical range can be set according to actual needs. The second preset numerical range can be set to a numerical range of more than 10 times the IO delay of the logical unit number, that is, when the IO delay of the computing node is greater than 11 times the IO delay of the logical unit number, it is judged that the difference between the IO delay of the computing node and the IO delay of the logical unit number is within the second preset numerical range.

[0060] In a preferred implementation of the embodiment of the present invention, after locating the IO fault at the front end in step 130, the method further includes:

[0061] Step 131: If the computing node is a virtual machine, then obtain the IO latency of other computing nodes corresponding to the same logical unit number as the computing node, the IO latency of other virtual machines corresponding to the same physical machine as the computing node, and the IO latency of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located; if it is determined that the IO latency of other computing nodes corresponding to the same logical unit number as the computing node is abnormal, the IO fault is located in the front-end and back-end Internet networks; if it is determined that the IO latency of other virtual machines corresponding to the same physical machine as the computing node is normal, and the IO latency of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located is normal, the IO fault is located in the computing node; if it is determined that the IO latency of other virtual machines corresponding to the same physical machine as the computing node is abnormal, and the IO latency of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located is normal, the IO fault is located in the computing node; if it is determined that the IO latency of other virtual machines corresponding to the same physical machine as the computing node is abnormal, and the IO latency of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located is normal, the IO fault is located in the physical machine where the computing node is located.

[0062] Among them, after locating the IO fault at the front end in step 130, it can be further determined whether the computing node is a virtual machine or a physical machine. If it is determined that the computing node is a virtual machine, step 131 is executed again. The IO delay of other computing nodes corresponding to the same logical unit number as the computing node, the IO delay of other virtual machines corresponding to the same physical machine as the computing node, and the IO delay of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located can be calculated by the moving average method. After obtaining the IO delay of other computing nodes corresponding to the same logical unit number as the computing node, if it is determined that the IO delay of other computing nodes corresponding to the same logical unit number as the computing node is abnormal, the IO fault is located in the interconnection network between the front end and the back end. When determining whether the IO latency of other computing nodes corresponding to the same logical unit number as the computing node is abnormal, the IO latency of each other computing node can be compared with the corresponding IO latency threshold, and the computing node whose IO latency is greater than the IO latency threshold is identified as a computing node with abnormal IO latency, and the proportion of computing nodes with abnormal IO latency among all other computing nodes is counted. If the proportion of computing nodes with abnormal IO latency among all other computing nodes exceeds a preset proportion, it is determined that the IO latency of other computing nodes corresponding to the same logical unit number as the computing node is abnormal.

[0063] Among them, after obtaining the IO latency of other virtual machines corresponding to the same physical machine as the computing node and the IO latency of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located, if it is determined that the IO latency of other virtual machines corresponding to the same physical machine as the computing node is normal, and it is determined that the IO latency of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located is normal, then the IO fault is located in the computing node. If it is determined that the IO latency of other virtual machines corresponding to the same physical machine as the computing node is abnormal, and it is determined that the IO latency of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located is normal, then the IO fault is located in the physical machine where the computing node is located. When judging whether the IO latency of other virtual machines corresponding to the same physical machine as the computing node is normal, the IO latency of each other virtual machine can be compared with the corresponding IO latency threshold, and the virtual machine whose IO latency is greater than the IO latency threshold is identified as a virtual machine with abnormal IO latency. If there is no virtual machine with abnormal IO latency among all other virtual machines, then the IO latency of other virtual machines corresponding to the same physical machine as the computing node is judged to be normal. If there is a virtual machine with abnormal IO latency among all other virtual machines, then the IO latency of other virtual machines corresponding to the same physical machine as the computing node is judged to be abnormal. When judging whether the IO latency of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located is normal, the IO latency of each other physical machine can be compared with the corresponding IO latency threshold, and the physical machine whose IO latency is greater than the IO latency threshold is identified as a physical machine with abnormal IO latency. If there is no physical machine with abnormal IO latency among all other physical machines, then the IO latency of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located is judged to be normal. If there is a physical machine with abnormal IO latency among all other physical machines, then the IO latency of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located is judged to be abnormal.

[0064] Step 132: If the computing node is a physical machine, then obtain the IO latency of other computing nodes corresponding to the same logical unit number as the computing node; if it is determined that the IO latency of other computing nodes corresponding to the same logical unit number as the computing node is abnormal, locate the IO fault in the front-end and back-end interconnection networks; if it is determined that the IO latency of other computing nodes corresponding to the same logical unit number as the computing node is normal, locate the IO fault in the computing node.

[0065] Among them, after locating the IO fault at the front end in step 130, it can be further determined whether the computing node is a virtual machine or a physical machine. If it is determined that the computing node is a physical machine, step 132 is executed. The IO delay of other computing nodes corresponding to the same logical unit number as the computing node can be calculated by the moving average method. After obtaining the IO delay of other computing nodes corresponding to the same logical unit number as the computing node, if it is determined that the IO delay of other computing nodes corresponding to the same logical unit number as the computing node is abnormal, the IO fault is located in the interconnection network between the front end and the back end; if it is determined that the IO delay of other computing nodes corresponding to the same logical unit number as the computing node is normal, the IO fault is located in the computing node. When determining whether the IO delay of other computing nodes corresponding to the same logical unit number as the computing node is normal, the IO delay of each other computing node can be compared with the IO delay threshold, and the computing node that makes the IO delay greater than the IO delay threshold will be identified as a computing node with abnormal IO delay. If a preset proportion of computing nodes in all other computing nodes corresponding to the same logical unit number as the computing node have abnormal IO latency, then the IO latency of other computing nodes corresponding to the same logical unit number as the computing node is judged to be abnormal, otherwise, the IO latency of other computing nodes corresponding to the same logical unit number as the computing node is judged to be normal, and the preset proportion may be, for example, 90%. The other computing nodes may be virtual machines or physical machines.

[0066] Step 140: If the difference between the IO latency of the logical unit number and the IO latency threshold of the logical unit number is within a third preset value range, and the difference between the IO latency of the computing node and the IO latency of the logical unit number is within a fourth preset value range, the IO fault is located at the back end.

[0067] If the difference between the IO delay of the logical unit number and the IO delay threshold of the logical unit number is determined to be within a third preset value range, the IO delay of the logical unit number is determined to be abnormal. The third preset value range can be set according to actual needs. For example, the third preset value range can be set to a value range of more than 10 times the IO delay threshold of the logical unit number, that is, when the IO delay of the logical unit number is greater than 11 times the IO delay threshold of the logical unit number, the difference between the IO delay of the logical unit number and the IO delay threshold of the logical unit number is determined to be within the third preset value range.

[0068] If it is determined that the difference between the IO latency of the computing node and the IO latency of the logical unit number is within a fourth preset numerical range, it is determined that the IO latency of the computing node is close to the IO latency of the logical unit number. The fourth preset numerical range can be set according to actual needs. For example, the fourth preset numerical range can be set to a numerical range of less than 0.1 times the IO latency of the logical unit number, that is, when the IO latency of the computing node is less than 1.1 times the IO latency of the logical unit number, it is determined that the difference between the IO latency of the computing node and the IO latency of the logical unit number is within the fourth preset numerical range.

[0069] In a preferred implementation of the embodiment of the present invention, after locating the IO fault at the backend in step 140, the method further includes:

[0070] Step 141: Obtain the ping network delay of each backend storage node, the average ping network delay of all backend storage nodes, and the ping network delay threshold of the backend storage nodes; if the difference between the average ping network delay of all backend storage nodes and the ping network delay threshold of the backend storage nodes is within the fifth preset value range, and the difference between the ping network delay of each backend storage node and the ping network delay threshold of the backend storage nodes is within the sixth preset value range, then locate the IO fault in the Internet network between the backend storage nodes; if the difference between the average ping network delay of all backend storage nodes and the ping network delay threshold of the backend storage nodes is within the seventh preset value range, and the ping network delay of a preset number of backend storage nodes is within the seventh preset value range, then locate the IO fault in the Internet network between the backend storage nodes; If the difference between the network delay and the ping network delay threshold of the back-end storage node is within the eighth preset value range, and the ping network delay of other back-end storage nodes outside the preset number of back-end storage nodes is not greater than the ping network delay threshold of the back-end storage node, the IO fault is located in the network of the preset number of back-end storage nodes; if the difference between the average value of the ping network delay of all back-end storage nodes and the ping network delay threshold of the back-end storage node is within the ninth preset value range, the IO delay of the built-in disk of all back-end storage nodes and the IO delay threshold of the built-in disk corresponding to the IO delay of the built-in disk are obtained, and the built-in disk whose IO delay and the IO delay threshold of the built-in disk are within the tenth preset value range will be identified as an abnormal built-in disk, and the IO fault will be located in the abnormal built-in disk.

[0071] Among them, the ping network delay of each backend storage node and the average ping network delay of all backend storage nodes can be calculated by the moving average method. If the difference between the average ping network delay of all backend storage nodes and the ping network delay threshold of the backend storage nodes is within the fifth preset numerical range, it is judged that the average ping network delay of all backend storage nodes is abnormal. The fifth preset numerical range can be set to a numerical range of more than 10 times the ping network delay threshold of the backend storage node. If the difference between the ping network delay of each backend storage node and the ping network delay threshold of the backend storage node is within the sixth preset numerical range, it is judged that the ping network delay of each backend storage node is abnormal. The sixth preset numerical range can be set to a numerical range of more than 10 times the ping network delay threshold of the backend storage node.

[0072] Among them, if the difference between the average value of the ping network delay of all backend storage nodes and the ping network delay threshold of the backend storage nodes is within the seventh preset numerical range, it is determined that the average value of the ping network delay of all backend storage nodes is abnormal. The seventh preset numerical range can be set to a numerical range of more than 10 times the ping network delay threshold of the backend storage nodes. If the difference between the ping network delay of a preset number of backend storage nodes and the ping network delay threshold of the backend storage nodes is within the eighth preset numerical range, it is determined that the ping network delay of the preset number of backend storage nodes is abnormal. The eighth preset numerical range can be set to a numerical range of more than 10 times the ping network delay threshold of the backend storage nodes. The preset number can be set to 1. If the difference between the average value of the ping network delay of all backend storage nodes and the ping network delay threshold of the backend storage nodes is within the ninth preset numerical range, it is determined that the average value of the ping network delay of all backend storage nodes is normal. The ninth preset numerical range can be set to a numerical range within 0.1 times the ping network delay threshold of the backend storage nodes. The tenth preset numerical range may be set to a numerical range greater than 10 times the IO latency threshold of the built-in disk, for example.

[0073] In an embodiment of the present invention, the IO delay of the front-end computing node and the IO delay threshold of the external disk of the computing node are obtained; if the difference between the IO delay of the computing node and the IO delay threshold of the external disk is within the first preset value range, the IO delay of the logical unit number corresponding to the computing node and the IO delay threshold of the logical unit number are further obtained; if the IO delay of the logical unit number is not greater than the IO delay threshold of the logical unit number, and the difference between the IO delay of the computing node and the IO delay of the logical unit number is within the second preset value range, the IO fault is located at the front end; if the difference between the IO delay of the logical unit number and the IO delay threshold of the logical unit number is within the third preset value range, and the difference between the IO delay of the computing node and the IO delay of the logical unit number is within the fourth preset value range, the IO fault is located at the back end. It can be seen that the embodiment of the present invention can quickly and accurately locate the IO fault by comparing the IO delay of the front-end computing node with the IO delay of the back-end logical unit number.

[0074] Figure 2 FIG. 2 shows a schematic diagram of the structure of an embodiment of an IO fault locating device of the present invention. Figure 2 As shown, the device 300 includes: a first acquisition module 310 , a second acquisition module 320 and a positioning module 330 .

[0075] A first acquisition module 310 is used to acquire the IO latency of the front-end computing node and the IO latency threshold of the external disk of the computing node;

[0076] A second acquisition module 320 is configured to acquire the IO latency of the logical unit number corresponding to the computing node and the IO latency threshold of the logical unit number if the difference between the IO latency of the computing node and the IO latency threshold of the external disk is within a first preset value range;

[0077] The positioning module 330 is used to locate the IO fault at the front end if the IO delay of the logical unit number is not greater than the IO delay threshold of the logical unit number, and the difference between the IO delay of the computing node and the IO delay of the logical unit number is within a second preset numerical range; if the difference between the IO delay of the logical unit number and the IO delay threshold of the logical unit number is within a third preset numerical range, and the difference between the IO delay of the computing node and the IO delay of the logical unit number is within a fourth preset numerical range, then locate the IO fault at the back end.

[0078] In an optional manner, the positioning module 330 is further used to, after executing the positioning of the IO fault at the front end, when the computing node is a virtual machine, respectively obtain the IO delays of other computing nodes corresponding to the same logical unit number as the computing node, the IO delays of other virtual machines corresponding to the same physical machine as the computing node, and the IO delays of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located;

[0079] If it is determined that the IO latency of other computing nodes corresponding to the same logical unit number as the computing node is abnormal, the IO fault is located in the interconnection network between the front end and the back end;

[0080] If it is determined that the IO latency of other virtual machines corresponding to the same physical machine as the computing node is normal, and if it is determined that the IO latency of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located is normal, the IO fault is located in the computing node;

[0081] If the IO latency of other virtual machines corresponding to the same physical machine as the computing node is determined to be abnormal, and the IO latency of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located is determined to be normal, the IO fault is located on the physical machine where the computing node is located.

[0082] In an optional manner, the positioning module 330 is further used to obtain the IO delay of other computing nodes corresponding to the same logical unit number as the computing node when the computing node is a physical machine after the IO fault is positioned at the front end;

[0083] If it is determined that the IO latency of other computing nodes corresponding to the same logical unit number as the computing node is abnormal, the IO fault is located in the interconnection network between the front end and the back end;

[0084] If it is determined that the IO delays of other computing nodes corresponding to the same logical unit number as the computing node are normal, the IO fault is located on the computing node.

[0085] In an optional manner, the positioning module 330 is further used to obtain the ping network delay of each backend storage node, the average ping network delay of all backend storage nodes, and the ping network delay threshold of the backend storage node after locating the IO fault at the backend;

[0086] If the difference between the average value of the ping network delay of all backend storage nodes and the ping network delay threshold of the backend storage nodes is within the fifth preset value range, and the difference between the ping network delay of each backend storage node and the ping network delay threshold of the backend storage node is within the sixth preset value range, the IO fault is located in the interconnection network between the backend storage nodes;

[0087] If the difference between the average value of the ping network delay of all backend storage nodes and the ping network delay threshold of the backend storage nodes is within the seventh preset value range, and the difference between the ping network delay of a preset number of backend storage nodes and the ping network delay threshold of the backend storage nodes is within the eighth preset value range, and the ping network delay of other backend storage nodes other than the preset number of backend storage nodes is not greater than the ping network delay threshold of the backend storage nodes, then the IO fault is located in the network of the preset number of backend storage nodes;

[0088] If the difference between the average value of the ping network delay of all back-end storage nodes and the ping network delay threshold of the back-end storage nodes is within the ninth preset value range, then the IO delay of the built-in disk of all back-end storage nodes and the IO delay threshold of the built-in disk corresponding to the IO delay of the built-in disk are obtained, and the built-in disk whose difference between the IO delay of the built-in disk and the IO delay threshold of the built-in disk is within the tenth preset value range will be identified as an abnormal built-in disk, and the IO fault will be located at the abnormal built-in disk.

[0089] In an embodiment of the present invention, the first acquisition module can acquire the IO delay of the front-end computing node and the IO delay threshold of the external disk of the computing node, and the second acquisition module can acquire the IO delay of the logical unit number corresponding to the computing node and the IO delay threshold of the logical unit number when the difference between the IO delay of the computing node and the IO delay threshold of the external disk is within a first preset value range, and the positioning module can compare the IO delay of the logical unit number with the IO delay threshold of the logical unit number, and compare the IO delay of the computing node with the IO delay of the logical unit number, so as to locate the IO fault at the front end or locate the IO fault at the back end. It can be seen that the IO fault positioning device of the embodiment of the present invention can quickly and accurately locate the IO fault.

[0090] Figure 3 The schematic diagram of the structure of the computing device embodiment of the present invention is shown. The specific embodiment of the present invention does not limit the specific implementation of the computing device.

[0091] like Figure 3 As shown, the computing device may include: a processor (processor) 402 , a communications interface (Communications Interface) 404 , a memory (memory) 406 , and a communication bus 408 .

[0092] The processor 402, the communication interface 404, and the memory 406 communicate with each other via the communication bus 408. The communication interface 404 is used to communicate with other devices such as a client or other server network elements. The processor 402 is used to execute the program 410, which can specifically execute the relevant steps in the above-mentioned embodiment of the method for IO fault location.

[0093] Specifically, the program 410 may include program code including computer executable instructions.

[0094] The processor 402 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the computing device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0095] The memory 406 is used to store the program 410. The memory 406 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0096] The program 410 may be specifically invoked by the processor 402 to cause the computing device to perform the following operations:

[0097] Obtaining the IO latency of the front-end computing node and the IO latency threshold of the external disk of the computing node;

[0098] If the difference between the IO latency of the computing node and the IO latency threshold of the external disk is within a first preset value range, further obtaining the IO latency of the logical unit number corresponding to the computing node and the IO latency threshold of the logical unit number;

[0099] If the IO latency of the logical unit number is not greater than the IO latency threshold of the logical unit number, and the difference between the IO latency of the computing node and the IO latency of the logical unit number is within a second preset value range, the IO fault is located at the front end;

[0100] If the difference between the IO latency of the logical unit number and the IO latency threshold of the logical unit number is within a third preset value range, and the difference between the IO latency of the computing node and the IO latency of the logical unit number is within a fourth preset value range, the IO fault is located at the back end.

[0101] In an optional manner, the program 410 is called by the processor 402 to enable the computing device to perform the following operations:

[0102] When the computing node is a virtual machine, the IO latency of other computing nodes corresponding to the same logical unit number as the computing node, the IO latency of other virtual machines corresponding to the same physical machine as the computing node, and the IO latency of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located are respectively obtained;

[0103] If it is determined that the IO delay of other computing nodes corresponding to the same logical unit number as the computing node is abnormal, the IO fault is located in the interconnection network between the front end and the back end;

[0104] If it is determined that the IO latency of other virtual machines corresponding to the same physical machine as the computing node is normal, and it is determined that the IO latency of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located is normal, the IO fault is located at the computing node;

[0105] If it is determined that the IO latency of other virtual machines corresponding to the same physical machine as the computing node is abnormal, and if it is determined that the IO latency of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located is normal, the IO fault is located on the physical machine where the computing node is located.

[0106] In an optional manner, the program 410 is called by the processor 402 to enable the computing device to perform the following operations:

[0107] When the computing node is a physical machine, obtaining the IO latency of other computing nodes corresponding to the same logical unit number as the computing node;

[0108] If it is determined that the IO delay of other computing nodes corresponding to the same logical unit number as the computing node is abnormal, the IO fault is located in the interconnection network between the front end and the back end;

[0109] If it is determined that the IO delays of the other computing nodes corresponding to the same logical unit number as the computing node are normal, the IO fault is located at the computing node.

[0110] In an optional manner, the program 410 is called by the processor 402 to enable the computing device to perform the following operations:

[0111] Get the ping network delay of each backend storage node, the average ping network delay of all backend storage nodes, and the ping network delay threshold of the backend storage nodes;

[0112] If the difference between the average value of the ping network delay of all the back-end storage nodes and the ping network delay threshold of the back-end storage node is within the fifth preset value range, and the difference between the ping network delay of each back-end storage node and the ping network delay threshold of the back-end storage node is within the sixth preset value range, the IO fault is located in the Internet network between the back-end storage nodes;

[0113] If the difference between the average value of the ping network delay of all the back-end storage nodes and the ping network delay threshold of the back-end storage nodes is within the seventh preset value range, and the difference between the ping network delay of a preset number of back-end storage nodes and the ping network delay threshold of the back-end storage nodes is within the eighth preset value range, and the ping network delay of other back-end storage nodes other than the preset number of back-end storage nodes is not greater than the ping network delay threshold of the back-end storage nodes, then the IO fault is located in the network of the preset number of back-end storage nodes;

[0114] If the difference between the average value of the ping network delay of all the back-end storage nodes and the ping network delay threshold of the back-end storage nodes is within a ninth preset numerical range, the IO delay of the built-in disks of all the back-end storage nodes and the IO delay threshold of the built-in disks corresponding to the IO delay of the built-in disks are obtained, and the built-in disks whose difference between the IO delay of the built-in disk and the IO delay threshold of the built-in disk is within a tenth preset numerical range are identified as abnormal built-in disks, and the IO fault is located at the abnormal built-in disk.

[0115] In an embodiment of the present invention, the program in the computing device is called by the processor to enable the computing device to execute the acquisition of the IO delay of the front-end computing node and the IO delay threshold of the external disk of the computing node; if the difference between the IO delay of the computing node and the IO delay threshold of the external disk is within the first preset value range, then the IO delay of the logical unit number corresponding to the computing node and the IO delay threshold of the logical unit number are further acquired; if the IO delay of the logical unit number is not greater than the IO delay threshold of the logical unit number, and the difference between the IO delay of the computing node and the IO delay of the logical unit number is within the second preset value range, then the IO fault is located at the front end; if the difference between the IO delay of the logical unit number and the IO delay threshold of the logical unit number is within the third preset value range, and the difference between the IO delay of the computing node and the IO delay of the logical unit number is within the fourth preset value range, then the IO fault is located at the back end. It can be seen that the computing device can quickly and accurately locate the IO fault.

[0116] Figure 4 FIG. 1 is a schematic diagram showing a structure of an embodiment of an IO fault location device of the present invention. Figure 4As shown, the device 500 includes:

[0117] The collection module 510 is used to collect IO delay information.

[0118] Among them, the IO latency information may include: the IO latency of the front-end computing node, the IO latency of the logical unit number corresponding to the computing node, the ping network latency of each back-end storage node, and, when the computing node is a virtual machine, the IO latency of other computing nodes corresponding to the same LUN as the computing node, the IO latency of other virtual machines corresponding to the same physical machine as the computing node, the IO latency of other physical machines corresponding to the same LUN as the physical machine where the computing node is located, and, when the current computing node is a physical machine, the IO latency of other computing nodes corresponding to the same LUN as the computing node.

[0119] The configuration module 520 is used to configure the IO latency threshold.

[0120] Among them, the IO latency threshold may include: the IO latency threshold of the external disk corresponding to the computing node, the IO latency threshold of the logical unit number corresponding to the computing node, the ping network latency threshold of the back-end storage node, and, when the computing node is a virtual machine, the IO latency threshold for configuring other computing nodes corresponding to the same logical unit number as the computing node, the IO latency threshold of other virtual machines corresponding to the same physical machine as the computing node, the IO latency threshold of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located, and, when the current computing node is a physical machine, the IO latency threshold for configuring other computing nodes corresponding to the same logical unit number as the computing node.

[0121] The positioning module 530 is used to execute the above-mentioned IO fault positioning method to locate the IO fault.

[0122] The display module 540 is used to obtain the result of the positioning module on the IO fault and display the result.

[0123] In an optional manner, the collection module 510 is also used to collect IOPS information, where IOPS stands for Input / Output Per Second and refers to the input and output volume of the disk per second. The collection module can collect IOPS information of the storage disk of the backend storage node.

[0124] In an optional manner, the display module 540 is also used to display the IO latency information collected by the collection module 510, and to display the correspondence between the external disk of the front-end computing node and the logical unit number of the back-end storage cluster.

[0125] In an optional manner, the configuration module 520 is further configured to:

[0126] Acquire the configuration information of the front-end computing node and the configuration information of the back-end storage cluster, and send the configuration information of the front-end computing node and the configuration information of the back-end storage cluster to the positioning module 530;

[0127] The positioning module 530 is further used to locate the IO fault according to the configuration information of the front-end computing node and the configuration information of the back-end storage cluster.

[0128] Among them, the configuration information of the front-end computing node may include: the host name of the computing node, the external disk information of the computing node, and the storage cluster information corresponding to the computing node; the configuration information of the back-end storage cluster may include: the name of the storage cluster corresponding to the computing node, the number of storage nodes contained in the storage cluster corresponding to the computing node and the name of each storage node, the number of logical unit numbers divided by the storage cluster corresponding to the computing node, and the node information of other computing nodes corresponding to the same storage cluster as the computing node.

[0129] In an optional manner, the configuration module 520 is further used to obtain the correspondence between the upper and lower layers of the backend storage cluster, and send the obtained correspondence between the upper and lower layers of the backend storage cluster to the positioning module 530;

[0130] The positioning module 530 is further used to locate the IO fault according to the corresponding relationship between the upper and lower layers of the backend storage cluster.

[0131] In an optional manner, the configuration module 520 is also used to obtain reference information and configure the IO latency threshold based on the reference information. The reference information may include: the hardware configuration of the computing node, the hardware configuration of the backend storage node corresponding to the computing node, and the IO latency requirements of the tasks processed by the computing node.

[0132] In the embodiment of the present invention, the collection module can collect IO delay information, the configuration module can configure the IO delay threshold, the positioning module can execute the IO fault positioning method to locate the IO fault, and the display module can obtain the result of the positioning module locating the IO fault and display the result. It can be seen that the IO fault positioning device of the embodiment of the present invention can quickly and accurately locate the IO fault, and display the IO fault positioning result to facilitate further analysis of the IO fault.

[0133] An embodiment of the present invention provides a computer-readable storage medium storing at least one executable instruction. When the executable instruction is executed on an IO fault locating device, the IO fault locating device executes the IO fault locating method in any of the above method embodiments.

[0134] An embodiment of the present invention provides a computer program, which can be called by a processor to enable a computing device to execute the IO fault locating method in any of the above method embodiments.

[0135] An embodiment of the present invention provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed on a computer, the computer executes the IO fault locating method in any of the above method embodiments.

[0136] The algorithm or display provided herein is not inherently related to any particular computer, virtual system or other equipment. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious to construct the structure required for this type of system. In addition, the embodiment of the present invention is not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description made to specific languages ​​above is for disclosing the best mode of the present invention.

[0137] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0138] Similarly, it should be understood that in order to streamline the present invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the embodiments of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this method of disclosure should not be interpreted as reflecting an intention that the claimed invention requires more features than those expressly recited in each claim.

[0139] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0140] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be understood as limitations on the order of execution.

Claims

1. A method for locating an IO fault, characterized in that: The IO failure is an IO failure of distributed storage, and the method includes: Obtaining the IO latency of the front-end computing node and the IO latency threshold of the external disk of the computing node; If the difference between the IO latency of the computing node and the IO latency threshold of the external disk is within a first preset value range, further obtaining the IO latency of the logical unit number corresponding to the computing node and the IO latency threshold of the logical unit number; If the IO latency of the logical unit number is not greater than the IO latency threshold of the logical unit number, and the difference between the IO latency of the computing node and the IO latency of the logical unit number is within a second preset value range, the IO fault is located at the front end; If the computing node is a virtual machine, the IO delays of other computing nodes corresponding to the same logical unit number as the computing node, the IO delays of other virtual machines corresponding to the same physical machine as the computing node, and the IO delays of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located are respectively obtained; if it is determined that the IO delays of other computing nodes corresponding to the same logical unit number as the computing node are abnormal, the IO fault is located in the front-end and back-end Internet networks; if it is determined that the IO delays of other virtual machines corresponding to the same physical machine as the computing node are normal, and it is determined that the IO delays of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located are normal, the IO fault is located in the computing node; if it is determined that the IO delays of other virtual machines corresponding to the same physical machine as the computing node are abnormal, and it is determined that the IO delays of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located are normal, the IO fault is located in the physical machine where the computing node is located; If the difference between the IO latency of the logical unit number and the IO latency threshold of the logical unit number is within a third preset value range, and the difference between the IO latency of the computing node and the IO latency of the logical unit number is within a fourth preset value range, the IO fault is located at the back end.

2. The method according to claim 1, characterized in that After locating the IO fault at the back end, the method further comprises: Get the ping network delay of each backend storage node, the average ping network delay of all backend storage nodes, and the ping network delay threshold of the backend storage nodes; If the difference between the average value of the ping network delay of all the back-end storage nodes and the ping network delay threshold of the back-end storage node is within the fifth preset value range, and the difference between the ping network delay of each back-end storage node and the ping network delay threshold of the back-end storage node is within the sixth preset value range, the IO fault is located in the Internet network between the back-end storage nodes; If the difference between the average value of the ping network delay of all the back-end storage nodes and the ping network delay threshold of the back-end storage nodes is within the seventh preset value range, and the difference between the ping network delay of a preset number of back-end storage nodes and the ping network delay threshold of the back-end storage nodes is within the eighth preset value range, and the ping network delay of other back-end storage nodes other than the preset number of back-end storage nodes is not greater than the ping network delay threshold of the back-end storage nodes, then the IO fault is located in the network of the preset number of back-end storage nodes; If the difference between the average value of the ping network delay of all the back-end storage nodes and the ping network delay threshold of the back-end storage nodes is within a ninth preset numerical range, the IO delay of the built-in disks of all the back-end storage nodes and the IO delay threshold of the built-in disks corresponding to the IO delay of the built-in disks are obtained, and the built-in disks whose difference between the IO delay of the built-in disk and the IO delay threshold of the built-in disk is within a tenth preset numerical range are identified as abnormal built-in disks, and the IO fault is located at the abnormal built-in disk.

3. An IO fault locating device, characterized in that: The IO failure is an IO failure of distributed storage, and the device includes: A first acquisition module is used to acquire the IO latency of the front-end computing node and the IO latency threshold of the external disk of the computing node; A second acquisition module is used to acquire the IO latency of the logical unit number corresponding to the computing node and the IO latency threshold of the logical unit number if the difference between the IO latency of the computing node and the IO latency threshold of the external disk is within a first preset value range; a positioning module, for locating the IO fault at the front end if the IO delay of the logical unit number is not greater than the IO delay threshold of the logical unit number, and the difference between the IO delay of the computing node and the IO delay of the logical unit number is within a second preset value range; if the computing node is a virtual machine, then respectively obtaining the IO delays of other computing nodes corresponding to the same logical unit number as the computing node, the IO delays of other virtual machines corresponding to the same physical machine as the computing node, and the IO delays of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located; if it is determined that the IO delays of the other computing nodes corresponding to the same logical unit number as the computing node are abnormal, then locating the IO fault in the Internet network between the front end and the back end; if it is determined that the IO delays of the other computing nodes corresponding to the same logical unit number as the computing node are abnormal, then locating the IO fault in the Internet network between the front end and the back end; If the IO latency of other virtual machines of a physical machine is normal, and it is judged that the IO latency of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located is normal, the IO fault is located in the computing node; if it is judged that the IO latency of other virtual machines corresponding to the same physical machine as the computing node is abnormal, and it is judged that the IO latency of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located is normal, the IO fault is located in the physical machine where the computing node is located; if the difference between the IO latency of the logical unit number and the IO latency threshold of the logical unit number is within a third preset numerical range, and the difference between the IO latency of the computing node and the IO latency of the logical unit number is within a fourth preset numerical range, the IO fault is located at the back end.

4. A computing device, characterized in that include: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform the operation of the IO fault locating method as described in any one of claims 1-2.

5. An IO fault location device, characterized in that: The IO failure is an IO failure of distributed storage, and the device includes: Collection module, used to collect IO delay information; Configuration module, used to configure IO latency threshold; A positioning module, used to execute the IO fault positioning method according to any one of claims 1 to 2 to locate the IO fault; The display module is used to obtain the result of the positioning module on the IO fault and display the result.

6. The device according to claim 5, characterized in that The display module is also used for: The IO latency information collected by the collection module is displayed, and the corresponding relationship between the external disk of the front-end computing node and the logical unit number of the back-end storage cluster is displayed.

7. The device according to claim 5 or 6, characterized in that The configuration module is also used to: Acquire the configuration information of the front-end computing node and the configuration information of the back-end storage cluster, and send the configuration information of the front-end computing node and the configuration information of the back-end storage cluster to the positioning module; The positioning module is also used to locate the IO fault according to the configuration information of the front-end computing node and the configuration information of the back-end storage cluster.

8. A computer-readable storage medium, characterized in that: The storage medium stores at least one executable instruction. When the executable instruction is executed on a computing device, the computing device executes the operation of the IO fault locating method according to any one of claims 1 to 2.

Images