Storage system fault root cause positioning method and device, computing equipment and storage medium

Abstract

The invention discloses a storage system fault root cause positioning method and device, computing equipment and a storage medium, and belongs to the technical field of computers. The method comprises the following steps: firstly, displaying a first node relation tree corresponding to a service node in which IO occurs in a storage system; based on the first node relation tree, a second node relation tree is displayed, and the second node relation tree indicates the data flow direction between the service nodes which have IO and are related to the fault; and then determining a fault root cause node based on the performance data of each service node in the second node relation tree. In the method, the content indicated by the second node relation tree and the basis (performance data) for determining the fault root cause node are matched with the content concerned by the operation and maintenance personnel of the storage system when the operation and maintenance personnel process the fault, and the determined fault root cause node can directly provide guidance for the operation and maintenance personnel to process the fault; and the operation and maintenance personnel do not need to further analyze the fault root cause node, so that the fault positioning efficiency and accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, computing device, and storage medium for locating the root cause of a storage system failure. Background Art

[0002] Storage systems have experienced tremendous growth in scale, business complexity, and performance service diversity, placing increasing demands on their reliability. Therefore, once a storage system experiences performance anomalies or business failures, it is necessary to efficiently and accurately locate the root cause of the failure, enabling storage system operations and maintenance personnel to quickly resolve the anomaly or business failure.

[0003] In related technologies, the root cause of a storage system fault is located by constructing a fault tree. The fault tree is constructed based on the logical relationship between various possible fault events that may occur in the storage system. The fault event that occurs in the storage system is taken as the top event, that is, the root of the fault tree. The fault event that can directly lead to the top event is taken as the first-level leaf node of the fault tree. The various fault events that can directly lead to the first-level leaf nodes are taken as the second-level leaf nodes of the fault tree. This process is repeated until a complete fault tree is obtained. Starting from the root node, the fault tree is analyzed layer by layer. First, it is determined whether the fault events corresponding to the various leaf nodes in the first layer have occurred. If the fault event corresponding to leaf node A has occurred, the fault event corresponding to leaf node A is the first-level intermediate cause of the fault. The fault event corresponding to the leaf nodes in the second layer related to leaf node A is determined to determine whether the fault events corresponding to the leaf nodes in the second layer have occurred. This determines the second-level intermediate cause of the fault. This process is repeated until the root cause of the fault is determined from the last layer of leaf nodes.

[0004] However, in the above method, operation and maintenance personnel need to determine one by one whether the fault event corresponding to the leaf node has occurred, which increases the manpower and time costs of locating the root cause of the fault, and the efficiency of locating the root cause of the fault is low. In addition, the granularity of the root cause of the fault located by the fault tree is at the event level. Operation and maintenance personnel need to further analyze the root cause of the fault at the event level to determine the specific root cause within the event. The efficiency and accuracy of root cause location are low. Summary of the Invention

[0005] The present invention provides a method, apparatus, computing device, and storage medium for locating the root cause of a storage system fault, which can improve the efficiency and accuracy of locating the root cause of a fault. The technical solution is as follows.

[0006] In a first aspect, a method for locating a root cause of a storage system failure is provided, the method comprising:

[0007] First, the first node relationship tree corresponding to the business node where IO (read and write) occurs in the storage system is displayed; then, based on the first node relationship tree, the second node relationship tree is displayed, which indicates the data flow between the business nodes where IO occurs and is related to the fault; then, based on the performance data of each business node in the second node relationship tree, the root cause node of the fault is determined in the second node relationship tree.

[0008] Among them, the business node is the business node where IO occurs in the storage system, that is, the processing logic for IO instructions in the storage system, for example, the function or method in the program code in the storage system for executing IO instructions such as read instructions or write instructions. In the storage system, the processing logic of IO instructions is in a tree-like nested relationship, that is, the processing logic of an IO instruction may include multiple sub-processing logics, or the processing logic of an IO instruction may call the processing logic of another IO instruction. Correspondingly, the granularity of the business node can be a business module, a function in a business module, or a sub-function of a function in a business module, etc. The granularity of the business node can be determined according to actual needs. The embodiment of the present application does not limit the granularity of the business node.

[0009] In the above method, the content indicated by the second node relationship tree and the basis for determining the root cause node of the fault (performance data) are consistent with the content that the storage system operation and maintenance personnel are concerned about when handling faults. The determined root cause node of the fault can directly provide guidance to the operation and maintenance personnel for handling the fault without the need for the operation and maintenance personnel to further analyze the root cause node of the fault, thereby improving the efficiency and accuracy of fault location.

[0010] Optionally, the first fault-related business nodes included in the second node relationship tree are business nodes that respond to the first read and write requests in the first node relationship tree, and the data flow between the first fault-related business nodes in the second node relationship tree is used to indicate the response order of the first fault-related business nodes to the first read and write requests.

[0011] The first read / write request carries a first tag, and the first tag is used to instruct the storage system to collect processing responses of the read / write request carrying the first tag, and report the service node corresponding to the collected processing response to the terminal.

[0012] Among them, the terminal determines the data flow direction between the business nodes related to the first fault based on the reception order of the processing responses to the first read and write request, including: for two processing responses with adjacent reception orders, the terminal uses the business node corresponding to the processing response received first as the source business node, and the business node corresponding to the processing response received later as the destination business node, and the data flow direction is from the source business node to the destination business node.

[0013] Optionally, the first node relationship tree further indicates the data flow between the business nodes where IO occurs in the storage system. The data flow between the business nodes in the first node relationship tree is used to represent the calling relationship between the business nodes indicated by the storage system code script.

[0014] Among them, between any two service nodes with a calling relationship, data flows from the called service node to the calling service node, that is, the called service node is the source service node and the calling service node is the destination service node.

[0015] Optionally, the second node relationship tree is a candidate node relationship tree that best matches a historical fault node relationship tree in the historical fault database among multiple candidate node relationship trees in the first node relationship tree.

[0016] Among them, after determining the most matching candidate node relationship tree, the number of times each historical fault node relationship tree in the historical fault database is successfully matched is recorded, so as to determine the data flow path with the most failures in the storage system, which is conducive to discovering the weak reliability links in the storage system, and then being able to perform targeted updates and iterations on the weak reliability links to improve the reliability of the storage system.

[0017] Optionally, the fault root cause node is determined based on the performance correlation between candidate fault root cause nodes in the second node relationship tree, and the candidate fault root cause node is a business node in a third node relationship tree in the second node relationship tree. The third node relationship tree is a node relationship tree with the highest-level fault business node in the second node relationship tree as the root node, and the fault business node is a business node in the second node relationship tree whose performance data is not within the target range.

[0018] The target range is the range of performance data of the service node under normal working conditions.

[0019] Optionally, the fault root cause node is a candidate fault root cause node in the third node relationship tree, whose performance correlation with the service node of the previous level is greater than a preset value and whose performance correlation with the service node of the next level is less than the preset value.

[0020] Among them, the process of determining the root cause node of the fault based on performance correlation can be understood as: starting from the business node with abnormal performance data at the top level, the business nodes whose performance correlation with the source business node is greater than or equal to the preset value are screened out in sequence along the data flow, until a business node whose performance correlation with the source business node is less than the preset value is found. The source business node of this business node is the root cause node of the fault.

[0021] Optionally, the method further includes:

[0022] Based on the first node relationship tree, a fourth node relationship tree of the storage system is displayed, where the fourth node relationship tree includes the business nodes related to the second fault in the business node where the IO occurs and the data flow between the business nodes related to the second fault; based on the performance data of each business node in the fourth node relationship tree, the root cause node of the second fault is determined in the fourth node relationship tree.

[0023] In the above method, different node relationship trees can be constructed based on different fault-related business nodes in the same first node relationship tree according to different faults. The operation and maintenance personnel do not need to rebuild the initial node relationship tree (first node relationship tree) for each fault, which can save labor costs and time costs, improve the efficiency of building the node relationship tree, and thus improve the efficiency of locating the root cause of storage system faults.

[0024] Optionally, the method further includes:

[0025] Receive a node relationship tree configuration instruction from a user, where the node relationship tree configuration instruction is used to adjust at least one of the service nodes, performance data of the service nodes, and data flow between the service nodes in the second node relationship tree; and adjust the second node relationship tree based on the node relationship tree configuration instruction.

[0026] In the above method, various components of the node relationship tree can be manually adjusted, thereby adjusting inappropriate business nodes, performance data or data flows in the node relationship tree, which can improve the accuracy of the node relationship tree and the flexibility of use of the node relationship tree.

[0027] In a second aspect, a storage system fault root cause location device is provided, which includes at least one functional module, and the at least one functional module is used to execute the storage system fault root cause location method provided by the first aspect or any possible implementation of the first aspect.

[0028] In a third aspect, a computing device is provided, the computing device comprising a processor and a memory;

[0029] The processor is configured to execute instructions stored in the memory, so that the computing device executes the storage system fault root cause location method provided in the first aspect or any optional manner of the first aspect.

[0030] In a fourth aspect, a computing device cluster is provided, comprising at least one computing device, each computing device comprising a processor and a memory;

[0031] The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the storage system fault root cause locating method provided in the first aspect or any optional manner of the first aspect.

[0032] In a fifth aspect, a computer-readable storage medium is provided, comprising computer program instructions. When the computer program instructions are executed by a computing device, the computing device executes the storage system fault root cause location method provided in the first aspect or any optional method in the first aspect.

[0033] In a sixth aspect, a computer-readable storage medium is provided, comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the storage system fault root cause location method provided in the first aspect or any optional method of the first aspect.

[0034] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Schematic diagram of an implementation environment of a storage system fault root cause location method provided by an embodiment of the present application;

[0036] Figure 2 This is a flow chart of a method for locating the root cause of a storage system failure provided by an embodiment of the present application;

[0037] Figure 3 is a schematic diagram of a first node relationship tree provided in an embodiment of the present application;

[0038] Figure 4 is a schematic diagram of a second node relationship tree provided in an embodiment of the present application;

[0039] Figure 5 is a schematic diagram of a second node relationship tree provided in an embodiment of the present application;

[0040] Figure 6 is a schematic diagram of a third node relationship tree provided in an embodiment of the present application;

[0041] Figure 7 This is a flowchart of locating the root cause of a storage system failure provided by an embodiment of the present application;

[0042] Figure 8 This is a flow chart of a method for locating the root cause of a storage system failure provided by an embodiment of the present application;

[0043] Figure 9 This is a flow chart of a method for locating the root cause of a storage system failure provided by an embodiment of the present application;

[0044] Figure 10 This is a flow chart of a method for locating the root cause of a storage system failure provided by an embodiment of the present application;

[0045] Figure 11 This is a schematic diagram of the structure of a storage system fault location device provided in an embodiment of the present application;

[0046] Figure 12 is a structural diagram of a computing device provided in an embodiment of the present application;

[0047] Figure 13 is a schematic diagram of a computing device cluster provided in an embodiment of the present application;

[0048] Figure 14 This is a schematic diagram of a possible implementation of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0050] First, the implementation environment of this application is introduced.

[0051] Figure 1 FIG. 1 is a schematic diagram of an implementation environment of a storage system fault root cause location method provided by an embodiment of the present application, such as Figure 1 As shown, the implementation environment includes a terminal 101 and a server 102. The terminal 101 and the server 102 are directly or indirectly connected via a wired or wireless manner, which is not limited.

[0052] Among them, the terminal 101 is at least one of a desktop computer, a laptop, a mobile terminal, a smart phone, a tablet computer, a smart watch, a virtual reality terminal, an augmented reality terminal, a wireless terminal and a laptop computer. The terminal 101 has a communication function and can access a wired network or a wireless network. The terminal 101 can generally refer to one of a plurality of terminals, and the embodiment of the present application only uses the terminal 101 as an example. Those skilled in the art will know that the number of the above-mentioned terminals 101 can be more or less. Schematically, a target application is running on the terminal 101, and the target application provides a storage system fault root cause location function, can obtain the operating data of the storage system (for example, performance data), and perform fault diagnosis on the storage system to determine the root cause of the fault of the storage system; in some embodiments, the target application can also display the root cause of the fault, for example, display the root cause of the fault in a tree structure, and the embodiment of the present application does not limit this. In some embodiments, the server 102 obtains the operating data of the storage system, performs fault diagnosis on the storage system to determine the root cause of the fault in the storage system, and the terminal 101 displays the root cause of the fault determined by the server 102, which is not limited in the embodiments of the present application. In some embodiments, the target application is associated with a historical fault database, which is used to store data related to historical faults, which is not limited in the embodiments of the present application. Among them, the target application takes the form of a web application, a microprogram, a client or an application plug-in, etc., which is not limited in this. Among them, a microprogram refers to a program that relies on other applications to run, such as a small program.

[0053] In some embodiments, the target application provides storage business services, and users can use the target application to send read and write requests and other operation requests to the storage system to complete the storage business. For example, the terminal 101 can send a read and write request to the storage device in the storage system to the application server in the storage system; the application server sends instructions corresponding to the read and write request to the storage device in the storage system to implement the read and write operation on the storage device; the application server returns the execution result corresponding to the read and write request to the terminal 101. The storage system can be a centralized storage system or a distributed storage system, etc. In some embodiments, the storage system is a single computing device or storage device. The embodiments of the present application do not limit the storage system.

[0054] Among them, the server 102 can be an independent physical server, or a server cluster or distributed file system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The number of the above-mentioned servers 102 can be more or less, and the embodiments of the present application are not limited to this. Exemplarily, the server 102 provides background services for the terminal 101. For example, the server 102 obtains the operating data of the storage system (for example, performance data), performs fault diagnosis on the storage system to determine the root cause of the fault of the storage system, sends the root cause of the fault to the terminal 101, and the terminal 101 displays the root cause of the fault, for example, in a tree structure. The embodiments of the present application are not limited to this.

[0055] In some embodiments, the wired or wireless network uses standard communication technologies and / or protocols. The network is typically the Internet, but can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network, or any combination of virtual private networks. In some embodiments, technologies and / or formats including hypertext markup language (HTML), extensible markup language (XML), etc. are used to represent data exchanged over the network. In addition, conventional encryption technologies such as secure socket layer (SSL), transport layer security (TLS), virtual private network (VPN), and internet protocol security (IPsec) can be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can be used to replace or supplement the above-mentioned data communication technologies.

[0056] An embodiment of the present application provides a method for locating the root cause of a storage system fault, in which the first node relationship tree corresponding to the business node where IO (reading and writing) occurs in the storage system is first displayed; then, based on the first node relationship tree, a second node relationship tree is displayed, the second node relationship tree indicating the data flow between the business nodes where IO occurs and is related to the fault; then, based on the performance data of each business node in the second node relationship tree, the root cause node of the fault is determined. In the above method, the content indicated by the second node relationship tree and the basis for determining the root cause node of the fault (performance data) are consistent with the content that the storage system operation and maintenance personnel pay attention to when handling the fault. The determined root cause node of the fault can directly provide guidance for the operation and maintenance personnel to handle the fault without the need for the operation and maintenance personnel to further analyze the root cause node of the fault, thereby improving the efficiency and accuracy of fault location.

[0057] Among them, the embodiment of the present application provides two ways to construct the second node relationship tree. The first way is: based on the processing response of the business node where IO occurs in the storage system to the target IO, determine the data flow between the fault-related business node associated with the target IO and the fault-related business node, and then construct the second node relationship tree; the second way is: based on the fault node relationship tree corresponding to the IO that has historically failed in the storage system, determine the candidate node relationship tree that best matches the fault node relationship tree among multiple candidate node relationship trees in the first node relationship tree, and use the candidate node relationship tree as the second node relationship tree.

[0058] In some embodiments, when it is determined which IO the fault is associated with, that is, when it is determined in which IO processing flow the fault occurs, the first method mentioned above is used to construct the second node relationship tree. When it is not determined which IO the fault is associated with, that is, when it is not determined in which IO processing flow the fault occurs, the second method mentioned above is used to construct the second node relationship tree. Through this embodiment, both the accuracy and efficiency of locating the root cause of the fault can be guaranteed. In other embodiments, after a storage system fails, multiple different types of IOs are sequentially sent to the storage system. The first method mentioned above is used to construct a corresponding second node relationship tree for each IO, and then each second node relationship tree is analyzed to determine at least one root cause node in the storage system. Through this embodiment, a comprehensive fault diagnosis can be performed on the storage system. When there are multiple faults in the storage system, the root causes of multiple faults can be comprehensively located, and it is helpful to discover potential fault hazards in the storage system, thereby improving the reliability of the storage system. In some other embodiments, after a storage system fails, the second node relationship tree is constructed using the second method described above, and the number of times each historical fault node relationship tree in the historical fault database is successfully matched is recorded, thereby determining the data flow path with the most failures in the storage system. This is conducive to discovering the links with weak reliability in the storage system, and then enabling targeted updates and iterations to be performed on the links with weak reliability to improve the reliability of the storage system.

[0059] The following describes in detail the storage system fault root cause location methods based on the above two methods of constructing the second node relationship tree.

[0060] First, the first method of constructing the second node relationship tree is taken as an example for introduction. Figure 2 This is a flow chart of a method for locating the root cause of a storage system failure provided by an embodiment of the present application. Figure 2 As shown, taking the method executed by a terminal as an example, the method includes the following steps 201 to 207.

[0061] 201. The terminal obtains multiple service nodes where I / O occurs in the storage system.

[0062] Among them, the storage system provides an operation data collection function, which can collect the operation data of the business nodes in the storage system. For example, the storage system includes a plug-in or subsystem for providing the operation data collection function, and the embodiments of the present application do not limit this. Alternatively, the terminal or server associated with the storage system provides an operation data collection function, which can collect the operation data of the business nodes in the storage system. The embodiments of the present application do not limit the method of collecting operation data. In subsequent embodiments, only the operation data collection function provided by the storage system is used as an example for explanation. The same applies to other operation data collection methods, which will not be repeated. The business node where IO occurs in the storage system is also the business node for which the operation data is collected.

[0063] The operation data includes the processing response of the service node (such as the return value) and the performance data of the service node. The collected performance data of the service node may include the service node's latency data, different types of function return values, input / output operations per second (IOPS), execution times, etc. The embodiments of this application do not limit the performance data.

[0064] In some embodiments, by configuring a service node for the operation data collection function of the storage system, for example, by associating information such as the identification of the service node or the name of the service node with the operation data collection function of the storage system, the operation data of the service node is collected. This process can be understood as the service node registering a point in a passive manner. Correspondingly, the terminal obtains multiple service nodes configured by the operation data collection function of the storage system to obtain multiple service nodes where IO occurs in the storage system. In other embodiments, by implanting the node information collection code corresponding to the operation data collection function of the storage system into the service node, the service node implanted with the node information collection code is associated with the operation data collection function of the storage system to collect the operation data of the service node. This process can be understood as the service node registering a point in an active manner. Correspondingly, the terminal obtains multiple service nodes implanted with the node information collection code in the storage system to obtain multiple service nodes where IO occurs in the storage system. In some other embodiments, some service nodes in the storage system register points in a passive manner, and some service nodes register points in an active manner. The embodiments of the present application are not limited to this. In some embodiments, after the service node is associated with the operation data collection function of the storage system, the storage system reports the associated service node to the terminal, and the terminal obtains the service node reported by the storage system.

[0065] In the above method, after any business node is associated with the operation data collection function of the storage system, the storage system can collect the performance data of the associated business node, and the business node becomes the analysis object of the terminal in the process of determining the root cause of the storage system failure. By registering points for business nodes of different granularity, the root cause of the storage system failure can be located to different granularity, that is, the granularity of the root cause of the failure can be customized. The larger the granularity of the business node, the coarser the determined root cause of the failure, and the smaller the granularity of the business node, the finer the determined root cause of the failure, thereby meeting different root cause location requirements of the failure. Moreover, for an extensible storage system, after the storage system is iterated, a new node relationship tree can be generated by re-registering points for the business nodes in the iterated storage system, and the root cause of the storage system failure can be located based on the new node relationship tree, without the need for operation and maintenance personnel to manually re-construct the fault tree for the iterated storage system, which can save the manpower cost and time cost of locating the root cause of the failure.

[0066] 202. The terminal displays a first node relationship tree based on the multiple service nodes where the IO occurs.

[0067] Among them, the first node relationship tree includes multiple levels, and each level includes at least one business node where IO occurs. The zeroth level of the first node relationship tree, that is, the root node of the first node relationship tree, indicates the IO instruction initiator. The IO instruction initiator is, for example, the client of the storage system, which is not limited in the embodiment of the present application. In some embodiments, the initiator of the IO instruction is the terminal, that is, the terminal not only provides the function of locating the root cause of the storage system fault, but also provides storage business services, which is not limited in the embodiment of the present application. The level at which any business node in the first node relationship tree is located indicates the path length from the business node to the root node, that is, the distance from the IO instruction to the business node from the initiator. For example, after the IO instruction is issued by the initiator, it passes through business nodes A, B and C in sequence. The distances between business nodes A, B and C and the initiator are 1, 2 and 3 respectively. Then the levels at which business nodes A, B and C are located in the first node relationship tree are the first level, the second level and the third level respectively. It should be noted that the description of the levels in the node relationship tree here is only exemplary and is not limited to this embodiment of the present application.

[0068] In some embodiments, when the operation data collection function is associated with any business node, the storage system records the distance between the business node and the IO instruction initiator; the terminal obtains the business nodes associated with the operation data collection function and the distance between each business node and the IO instruction initiator from the storage system; based on the distance between the business node and the IO instruction initiator, the level corresponding to each business node is determined; the terminal generates a first node relationship tree based on the level corresponding to each business node, and displays the first node relationship tree.

[0069] In some embodiments, the terminal displays each business node in the first node relationship tree as a first shape, and the first shape can be a circle, a square, a triangle, etc., which is not limited in the embodiments of the present application. In other embodiments, the terminal displays each business node in the first node relationship tree as a first color, and the first color can be yellow, green, blue, etc. In some further embodiments, the terminal displays the information of each business node in the first node relationship tree, such as the business node identifier, the name of the business node, or the level at which the business node is located, or the performance data of the business node. It should be noted that the above introduction to the form in which the terminal displays the first node relationship tree is only exemplary, and those skilled in the art can determine the display form of the first node relationship tree according to actual needs, which is not limited in the embodiments of the present application.

[0070] Below through Figure 3 , an example is given to illustrate the first node relationship tree. Figure 3 This is a schematic diagram of a first node relationship tree provided in an embodiment of the present application, such as Figure 3 As shown, the first node relationship tree includes 7 levels, the zeroth level includes the root node S0, the first level includes the business node A, the second level includes the business node BD, the third level includes the business node EK, the fourth level includes the business node LP, the fifth level includes the business node QV, and the sixth level includes the business node W. Each business node is represented by a blank circle. It should be noted that Figure 3 This is only an example of the first node relationship tree, and the embodiment of the present application does not limit the display form of the first node relationship tree.

[0071] 203. The terminal receives processing responses from multiple service nodes in the first node relationship tree to the first read / write request.

[0072] The first read / write request is used to perform a read / write operation on the storage system. In some embodiments, the read / write request is sent to the storage system by a storage system client, which can be run on the terminal or on other terminals or servers, and this embodiment of the application does not limit this.

[0073] Wherein, the first read / write request carries a first tag, and the first tag is used to instruct the storage system to collect the processing response of the read / write request carrying the first tag, and report the business node corresponding to the collected processing response to the terminal. In some embodiments, when the business node in the first node relationship tree registers the point in a passive manner, the storage system collects the processing response of the associated business node to the read / write request, and the storage system reports the processing response of the first read / write request in the collected processing response to the terminal. In other embodiments, when the business node in the first node relationship tree registers the point in an active manner, the node information collection code implanted in the business node is used to process the read / write request (first read / write request) carrying the first tag. If any business node in the first node relationship tree receives and processes the first read / write request, the business node returns the processing response for the first read / write request to the storage system; the storage system receives the processing responses of multiple business nodes in the first node relationship tree for the first read / write request, and reports the processing responses of multiple business nodes for the first read / write request to the terminal.

[0074] In some embodiments, the first tag is associated with the first business of the storage system, that is, the first tag is carried in the read and write requests associated with the first business of the storage system. While processing the read and write requests, the storage system collects the processing response and performance data of the business node corresponding to the first business in real time and reports it to the terminal, so that the terminal can monitor the business nodes related to the first business in real time, which is conducive to timely discovering faults in the storage system, thereby timely handling the faults, and improving the stability and security of the storage system.

[0075] In other embodiments, the first tag is associated with a first business of the storage system. After a failure occurs in the first business, the storage system client sends a read / write request carrying the first tag to the storage system, collects processing responses and performance data of the business node corresponding to the first business, and reports it to the terminal, so that the terminal locates the root cause of the failure of the first business. When the storage system is operating normally, the terminal does not need to work, which can reduce the workload of the storage system and reduce the occupation of computing resources and storage resources.

[0076] 204. The terminal determines multiple service nodes in the first node relationship tree that send processing responses to the first read / write request as first fault-related service nodes.

[0077] If, while processing read / write requests, the storage system collects, in real time, the processing responses and performance data of the service nodes corresponding to the first service and reports them to the terminal, the first fault is a monitored fault that has not yet occurred and can be understood as a fault that may occur while the storage system is processing the first service. If, after the first service fails, the storage system client sends a read / write request carrying the first tag to the storage system, collects, and reports to the terminal the processing responses and performance data of the service nodes corresponding to the first service, the first fault is a fault that has already occurred in the storage system.

[0078] 205. The terminal determines the data flow direction between the service nodes related to the first fault based on the order of receiving the processing responses to the first read / write request.

[0079] The reception order is the order in which the storage system receives processing responses to the first read / write request. In some embodiments, the storage system reports each processing response to the first read / write request to the terminal once, so the order in which the terminal receives the processing responses is equal to the order in which the storage system receives the processing responses. In other embodiments, after receiving multiple processing responses to the first read / write request, the storage system reports them to the terminal once and reports the order in which the multiple processing responses were received to the terminal.

[0080] Among them, the terminal determines the data flow direction between the business nodes related to the first fault based on the receiving order of the processing responses to the first read and write request, including: in the processing responses to the first read and write request, for two processing responses with adjacent receiving orders, the terminal uses the business node corresponding to the processing response received first as the source business node, and the business node corresponding to the processing response received later as the destination business node, and the data flow direction is from the source business node to the destination business node.

[0081] 206. The terminal displays a second node relationship tree of the storage system based on the data flow between the service nodes related to the first fault. The second node relationship tree includes the service nodes related to the first fault among the service nodes where IO occurs and the data flow between the service nodes related to the first fault.

[0082] In some embodiments, the terminal displays the second node relationship tree based on the first node relationship tree. For example, the terminal highlights the business node related to the first fault in the first node relationship tree, and uses arrows to indicate the data flow between the business nodes related to the first fault on the first node relationship tree, wherein the arrows between any two business nodes related to the first fault point from the source business node to the destination business node. It should be noted that the above example is only an illustrative description of a possible way to display the second node relationship tree based on the first node relationship tree, and the embodiments of the present application do not limit this. Figure 4As shown, Figure 4 is a schematic diagram of a second node relationship tree provided in an embodiment of the present application, Figure 4 The first node relationship tree includes a service node AW (in Figure 4 In the service node AW, the service nodes related to the first fault are A, B, D, E, L, Q, I, J, N, P, U and W (in Figure 4 Black circles are used for illustration), and arrows are used to indicate the data flow between the service nodes related to the first fault.

[0083] In other embodiments, the terminal constructs the second node relationship tree based on the data flow between the service node related to the first fault and the service node related to the first fault; the terminal displays the second node relationship tree. The embodiment of the present application does not limit the display method of the second node relationship tree. Figure 5 As shown, Figure 5 This is a schematic diagram of a second node relationship tree provided in an embodiment of the present application. Figure 5 Including service nodes A, B, D, E, L, Q, I, J, N, P, U and W related to the first fault, arrows are used to indicate the data flow between the service nodes related to the first fault.

[0084] In some embodiments, the terminal receives a node relationship tree configuration instruction from a user for the second node relationship tree, the node relationship tree configuration instruction being used to adjust at least one of the service nodes, performance data of the service nodes, and data flows between the service nodes in the second node relationship tree; and the terminal adjusts the second node relationship tree based on the node relationship tree configuration instruction. This embodiment enables manual adjustment of various components of the node relationship tree, thereby adjusting inappropriate service nodes, performance data, or data flows in the node relationship tree, thereby improving the accuracy of the node relationship tree and the flexibility of its use.

[0085] 207. The terminal determines a root cause node of the first fault in the second node relationship tree based on the performance data of each service node in the second node relationship tree.

[0086] Among them, if the storage system collects the processing response and performance data of the business node corresponding to the first business in real time while processing the read and write requests and reports them to the terminal, then the first fault is a fault that has not yet occurred but is being monitored. It can be understood as a fault that may occur in the storage system during the processing of the first business. The terminal monitors each node in the second node relationship tree based on the performance data of each business node in the second node relationship tree to determine the root cause node that may cause the fault. If, after the first business fails, the storage system client sends a read and write request carrying the first tag to the storage system, and collects the processing response and performance data of the business node corresponding to the first business and reports them to the terminal, then the first fault is a fault that has already occurred in the storage system. The terminal determines the root cause node of the fault that has already occurred based on the performance data of each business node in the second node relationship tree.

[0087] The process of the terminal determining the root cause node of the first fault includes the following steps 2071 to 2073.

[0088] 2071. Based on performance data of each service node in the second node relationship tree, determine at least one faulty service node in the second node relationship tree, where the performance data of the faulty service node is not within a target range.

[0089] Performance data includes at least one of the following: service node latency data, function return values of different types, input / output operations per second (IOPS), and execution count. The target range is the range of performance data for the service node under normal operating conditions.

[0090] 2072. Based on the second node relationship tree, a third node relationship tree in the storage system is displayed, where the third node relationship tree is used to indicate candidate fault root cause nodes in the second node relationship tree. The third node relationship tree is a node relationship tree in the second node relationship tree with the target business node as the root node, and the target business node is the highest-level fault business node among the at least one fault business node in the second node relationship tree.

[0091] The third node relationship tree can be understood as a subtree in the second node relationship tree with the target service node as the root node, and each service node in the subtree is a candidate fault root cause node.

[0092] In some embodiments, the terminal displays the third node relationship tree based on the second node relationship tree. For example, the terminal highlights the candidate fault root cause nodes in the second node relationship tree, and highlights the data flow between the candidate fault root cause nodes. It should be noted that the above example is only an illustrative description of a possible way to display the third node relationship tree based on the second node relationship tree, and the embodiments of the present application do not limit this. In other embodiments, the terminal determines the subtree with the target business node as the root node in the second node relationship tree based on the target business node; the terminal displays the third node relationship tree. The embodiments of the present application do not limit the display method of the third node relationship tree.

[0093] In some embodiments, the terminal receives a node relationship tree configuration instruction from a user for a third node relationship tree, and the node relationship tree configuration instruction is used to adjust at least one of the business nodes, performance data of the business nodes, and data flow between business nodes in the third node relationship tree; the terminal adjusts the third node relationship tree based on the node relationship tree configuration instruction.

[0094] 2073. Determine the fault root cause node in the third node relationship tree based on the performance correlation between each candidate fault root cause node in the third node relationship tree, wherein the performance correlation between each candidate fault root cause node is determined based on performance data of each candidate fault root cause node.

[0095] In some embodiments, the terminal determines the performance correlation between each candidate root cause node based on the performance data of each candidate root cause node. The process includes: for any two candidate root cause nodes with a data flow relationship, respectively determining the correlation between the two candidate root cause nodes in each performance data; based on the preset weight of each performance data, fusing the correlation between the two candidate root cause nodes in each performance data to obtain the performance correlation between the two candidate root cause nodes. Among them, the correlation between the two candidate root cause nodes in any performance data can be determined by calculating the Pearson correlation coefficient (Pearson) between the two candidate root cause nodes in the performance data. It should be noted that those skilled in the art can determine the method for calculating the performance correlation between the two nodes according to the needs, and the embodiment of the present application does not limit the calculation method of the performance correlation. In some embodiments, the performance correlation between each candidate root cause node and other service nodes is described in the form of an association relationship matrix, and the embodiment of the present application does not limit this.

[0096] In which, the terminal determines the root cause node of the fault in the third node relationship tree based on the performance correlation between each candidate root cause node in the third node relationship tree, including: in the third node relationship tree, determining the candidate root cause node whose performance correlation with the business node of the previous level is greater than or equal to a preset value and whose performance correlation with the business node of the next level is less than the preset value as the root cause node of the fault. In some embodiments, when the terminal displays the third node relationship tree, the performance correlation between each candidate root cause node of the fault is displayed in the third node relationship tree. The embodiments of the present application are not limited to this. In other embodiments, the terminal uses different shapes or colors to display the business nodes whose performance data is within the target range, the business nodes whose performance data is not within the target range but are not the root cause nodes of the fault, and the root cause nodes of the fault in the third node relationship tree, so that the operation and maintenance personnel can quickly obtain the root cause nodes of the fault and the business nodes with abnormal performance data from the third node relationship tree, thereby improving the efficiency of fault handling.

[0097] Below through Figure 6 , the process shown in the above steps 2071 to 2073 is illustrated by way of example. Figure 6 This is a schematic diagram of a third node relationship tree provided in an embodiment of the present application, such as Figure 6 As shown, the third node relationship tree includes multiple candidate fault root cause nodes D, I, J, N, P, U and W. The performance correlation between D and I is 0.2, the performance correlation between D and J is 0.9, the performance correlation between J and N is 0.3, the performance correlation between J and P is 0.9, the performance correlation between P and U is 0.8, and the performance correlation between U and W is 0.3. The preset value is 0.7. Figure 6 In the figure, the performance correlation between business node U and business node P of the previous level is 0.8, which is greater than the preset value 0.7, and the performance correlation between business node U and business node W of the next level is 0.3, which is less than the preset value 0.7. Therefore, business node U is the root cause node of the fault. The above process can be understood as starting from the business node with abnormal performance data at the top level, and filtering out business nodes whose performance correlation with the source business node is greater than or equal to the preset value along the data flow, until a business node whose performance correlation with the source business node is less than the preset value is found. In this case, the source business node of this business node is the root cause node of the fault. It should be noted that Figure 6 The description is made by taking the example that the terminal displays the third node relationship tree separately from the second node relationship tree. The embodiment of the present application does not limit the display method of the third node relationship tree.

[0098] The fine-grained nature of the fault root cause node corresponds to the fine-grained nature of the service node. Depending on the fine-grained nature of the service node, the fault root cause node can point to a specific service logic or parameter of the storage system, for example. The present embodiment does not impose any restrictions on the fine-grained nature of the service node, nor does it impose any restrictions on the fine-grained nature of the fault root cause node.

[0099] It should be noted that the above steps 2071 to 2073 are explained by taking the display of the third node relationship tree in the process of determining the root cause node of the fault as an example. In some embodiments, the terminal does not display the third node relationship tree, but directly outputs the determined root cause node of the fault. The embodiments of the present application do not limit this.

[0100] It should be noted that the description of the process of determining the root cause node of the fault based on the performance data in the above step 207 is merely exemplary. In some embodiments, the root cause node of the first fault is determined based on the performance data of each service node related to the first fault using a time series data analysis method; in other embodiments, the performance data of each service node related to the first fault is processed using a machine learning model to obtain the root cause node of the first fault; in still other embodiments, the performance data of each service node related to the first fault is processed using a statistical learning algorithm to obtain the root cause node of the first fault. The embodiments of the present application are not limited to this.

[0101] Below through Figure 7 The process shown in the above steps 201 to 207 is described with an example. Figure 7 FIG. 1 is a flow chart of locating the root cause of a storage system failure provided by an embodiment of the present application. Figure 7 As shown, the terminal constructs a second node relationship tree; based on the performance data of each node in the second node relationship tree, the terminal determines the root cause node of the fault in the second node relationship tree. The process of the terminal constructing the second node relationship tree includes: the terminal obtains the business node where IO occurs in the storage system and displays the first node relationship tree; the terminal issues a read and write request with a tag related to the business according to the business; the terminal obtains the processing response of the business node to the read and write request with a tag, determines the business node related to the fault related to the business, and determines the data flow between the business nodes related to the fault based on the order of receiving the processing response. This process can be understood as clarifying the IO path of the business; the terminal associates the performance data collected by the storage system with the business node related to the fault. This process can be understood as filling the performance data on the business node related to the fault; the terminal displays the second node relationship tree.

[0102] In the above method, the content indicated by the second node relationship tree and the basis for determining the fault root cause node (performance data) are consistent with the content that storage system operation and maintenance personnel are concerned about when handling faults. The determined fault root cause node can directly provide fault handling guidance to the operation and maintenance personnel without the need for the operation and maintenance personnel to further analyze the fault root cause node, thereby improving the efficiency and accuracy of fault locating. Furthermore, based on the processing response of the service node generating the IO in the storage system to the target IO, the data flow between the fault-related service node associated with the target IO and the fault-related service node is determined, and then the second node relationship tree is constructed. While the storage system is operating, the terminal can monitor the service nodes related to the first service in real time based on the IO related to the first service processed by the storage system, which is conducive to timely discovery of faults in the first service, thereby promptly handling the fault and improving the stability and security of the storage system. Alternatively, after the first service fails, a test IO for the first service can be sent to the storage system, and the terminal can locate the root cause of the fault in the first service based on the test IO, which can reduce the workload of the terminal and reduce the usage of the terminal's computing and storage resources.

[0103] It should be noted that the above Figure 2 The embodiment shown is explained by taking the determination of the root cause node of the first fault as an example. In some embodiments, the terminal displays the fourth node relationship tree of the storage system based on the first node relationship tree. The fourth node relationship tree includes the business nodes related to the second fault in the business node where IO occurs and the data flow between the business nodes related to the second fault; the terminal determines the root cause node of the second fault in the fourth node relationship tree based on the performance data of each business node in the fourth node relationship tree. That is, in the storage system fault root cause location method provided by the embodiment of the present application, different node relationship trees can be constructed based on different fault-related business nodes in the same first node relationship tree according to different faults. The operation and maintenance personnel do not need to rebuild the initial node relationship tree (first node relationship tree) for each fault, which can save labor costs and time costs, improve the efficiency of building the node relationship tree, and thus improve the efficiency of locating the root cause of the storage system fault.

[0104] The second method of constructing the second node relationship tree is taken as an example for description below. Figure 8 This is a flow chart of a method for locating the root cause of a storage system failure provided by an embodiment of the present application. Figure 8 As shown, taking the method executed by a terminal as an example, the method includes the following steps 801 to 806.

[0105] 801. A terminal obtains multiple service nodes where IO occurs in a storage system.

[0106] This step 801 is similar to the above step 201 and will not be described in detail.

[0107] 802. The terminal obtains a code script of the storage system, and based on the code script, determines a call relationship between multiple service nodes where IO occurs, where the call relationship indicates a data flow direction between the multiple service nodes where IO occurs.

[0108] Among them, the terminal determines the code scripts of multiple business nodes associated with the operation data collection function of the storage system in the code script; based on the keywords related to function calls (or method calls) in the code scripts of the multiple business nodes, the calling relationship between the multiple business nodes is determined. Among them, between any two business nodes with a calling relationship, data flows from the called business node to the calling business node, that is, the called business node is the source business node, and the calling business node is the destination business node. It should be noted that the above description of determining the calling relationship between business nodes based on the code script is only exemplary. In some embodiments, a code analysis tool is run on the terminal, and the code analysis tool provides a code call relationship analysis function. The terminal processes the code script of the storage system through the code analysis tool to obtain the calling relationship between multiple business nodes. The embodiment of the present application does not limit the method for determining the calling relationship between business nodes.

[0109] 803. The terminal displays a first node relationship tree based on the calling relationship.

[0110] Based on this call relationship, the terminal determines the hierarchy level of each service node associated with the storage system's operational data collection function. The terminal uses arrows to indicate the data flow between the service nodes indicated by this call relationship. An arrow between any two service nodes points from the source service node to the destination service node. The process for determining the hierarchy level of a service node by the terminal is similar to the process for determining the hierarchy level of a service node in step 202 above and is not further described.

[0111] In some embodiments, the terminal displays each business node in the first node relationship tree as a first shape, and the first shape can be a circle, a square, a triangle, etc., which is not limited in the embodiments of the present application. In other embodiments, the terminal displays each business node in the first node relationship tree as a first color, and the first color can be yellow, green, blue, etc. In some further embodiments, the terminal displays the information of each business node in the first node relationship tree, such as the business node identifier, the name of the business node, or the level at which the business node is located, or the performance data of the business node. It should be noted that the above introduction to the form in which the terminal displays the first node relationship tree is only exemplary, and those skilled in the art can determine the display form of the first node relationship tree according to actual needs, which is not limited in the embodiments of the present application.

[0112] 804. The terminal determines multiple candidate node relationship trees in the first node relationship tree.

[0113] The terminal determines multiple data flow paths in the first node relationship tree starting from the root node and ending at a service node with no data outflow, and each data flow path is a candidate node relationship tree.

[0114] In some embodiments, the terminal merges the mergeable candidate node relationship trees in multiple candidate node relationship trees into one candidate node relationship tree. For example, the data flow direction of the candidate node relationship tree T1 is 1-2-3-4, and the data flow direction of the candidate node relationship tree T2 is 1-2-3-5, then the terminal can merge T1 and T2 into 1-2-3-4 / 5. It should be noted that the above-mentioned example of the corresponding candidate node relationship tree merging process is only exemplary, and the embodiments of the present application do not limit this.

[0115] In some embodiments, the terminal receives a user's selection instruction for a candidate node relationship tree, where the selection instruction is used to determine a candidate node relationship tree in the first node relationship tree; the terminal determines the node relationship tree indicated by the selection instruction as the candidate node relationship tree.

[0116] 805. The terminal matches multiple candidate node relationship trees with the historical fault node relationship trees in the historical fault database respectively, and displays the candidate node relationship tree that best matches the historical fault node relationship tree among the multiple candidate node relationship trees as a second node relationship tree. The second node relationship tree includes the business nodes related to the first fault in the business nodes where IO occurs and the data flow between the business nodes related to the first fault.

[0117] The terminal is associated with a historical fault database, which is used to store a historical fault node relationship tree. The historical fault node relationship tree can be understood as a historical fault IO path. The historical fault database can be a database in the storage system or a database outside the storage system, and this embodiment of the application does not limit this.

[0118] The terminal associates the performance data of the service nodes collected by the storage system with the service nodes. Each service node in the historical fault node relationship tree is associated with the performance data of that service node under historical fault conditions. The terminal matches multiple candidate node relationship trees associated with the performance data with the historical fault node relationship tree associated with the performance data, and determines the candidate node relationship tree that best matches the historical fault node relationship tree. It should be understood that the candidate node relationship tree that best matches the historical fault node relationship tree has a similar tree structure and similar performance data representation as the historical fault node relationship tree.

[0119] Among them, the terminal can use at least one of the multivariate analysis methods (such as Pearson correlation coefficient) or multiple linear regression methods to determine the matching degree between the candidate node relationship tree and the historical fault node relationship tree. Technical personnel in this field can use other methods to determine the matching degree according to actual needs, and the embodiments of this application do not limit this.

[0120] 806. The terminal determines a root cause node of the first fault in the second node relationship tree based on the performance data of each service node in the second node relationship tree.

[0121] Step 806 is similar to step 207 and will not be described in detail.

[0122] In some embodiments, the terminal determines the data flow path of the first fault root node in the second node relationship tree as a new fault node relationship tree, and stores the new fault node relationship tree in the historical fault database. In this method, updating the historical fault database with the node relationship tree corresponding to the most recently occurring fault can expand the historical fault database, thereby facilitating improved accuracy in matching candidate node relationship trees with historical fault node relationship trees.

[0123] In some embodiments, the terminal records the number of times each historical fault node relationship tree in the historical fault database is successfully matched, thereby determining the data flow path with the most failures in the storage system, which is conducive to discovering the weak reliability links in the storage system, and then being able to perform targeted updates and iterations on the weak reliability links to improve the reliability of the storage system.

[0124] The following combination Figure 9 and Figure 10The process shown in the above steps 801 to 806 is described with an example. Figure 9 This is a flowchart of a method for locating the root cause of a storage system failure provided by an embodiment of the present application. Figure 10 This is a flow chart of a method for locating the root cause of a storage system failure provided by an embodiment of the present application. Figure 9 As shown, the terminal constructs a second node relationship tree; based on the performance data of each node in the second node relationship tree, the terminal determines the root cause node of the fault in the second node relationship tree. The process of the terminal constructing the second node relationship tree includes: the terminal obtains the business node where IO occurs in the storage system, determines the call relationship between multiple business nodes based on the code script of the storage system, and displays the first node relationship tree. This process can be understood as determining the full IO path (corresponding to Figure 10 Step 1); the terminal associates the performance data collected by the storage system with the business node in the first node relationship tree. This process can be understood as filling the performance data on the business node in the first node relationship tree (corresponding to Figure 10 Step 2); The terminal matches the candidate node relationship tree in the first node relationship tree with the historical fault node relationship tree in the historical fault database, and the terminal determines the candidate node relationship tree with the highest matching degree as the second node relationship tree (corresponding to Figure 10 Step 3 in the previous step).

[0125] In the above method, the content indicated by the second node relationship tree and the basis for determining the root cause node of the fault (performance data) are consistent with the content that the storage system operation and maintenance personnel pay attention to when handling faults. The determined root cause node of the fault can directly provide guidance for the operation and maintenance personnel to handle the fault without the need for the operation and maintenance personnel to further analyze the root cause node of the fault, thereby improving the efficiency and accuracy of fault location; further, by recording the number of times each historical fault node relationship tree in the historical fault database is successfully matched, the data flow path with the most failures in the storage system is determined, which is conducive to discovering the links with weak reliability in the storage system, and then being able to perform targeted updates and iterations on the links with weak reliability to improve the reliability of the storage system.

[0126] It should be noted that the above Figure 2 and Figure 8The method embodiments shown are all introduced by taking the terminal as the execution subject as an example. In some embodiments, the method takes the server or server cluster as the execution subject, and the server or server cluster is configured with a display module; in some other embodiments, the method is executed by the terminal and the server interactively, or by the terminal and the server cluster interactively. For example, the terminal is responsible for obtaining the business nodes and performance data associated with the operation data acquisition function of the storage system, and sending the business nodes and performance data to the server. The server constructs a node relationship tree and determines the root cause node of the fault in the node relationship tree based on the performance data. The server sends the constructed node relationship tree and the root cause node of the fault to the terminal, and the terminal displays the node relationship tree and the root cause node. It should be noted that the above examples of the process of interaction between the terminal and the server or server cluster are only exemplary. Those skilled in the art can change the above examples according to actual needs. Figure 2 or Figure 8 The method flow shown is deployed on a terminal and a server (or a server cluster), which is not limited in the embodiments of the present application.

[0127] Figure 11 FIG. 1 is a schematic diagram of a storage system fault location device provided in an embodiment of the present application. Figure 11 As shown, the device includes: a display module 1101 and a determination module 1102.

[0128] The display module 1101 is configured to display a first node relationship tree of the storage system, where the first node relationship tree indicates a service node in which IO occurs in the storage system.

[0129] The determining module 1102 is configured to display a second node relationship tree of the storage system based on the first node relationship tree, where the second node relationship tree includes service nodes related to the first fault among the service nodes where the IO occurs and data flows between the service nodes related to the first fault;

[0130] Based on the performance data of each service node in the second node relationship tree, a fault root cause node of the first fault is determined in the second node relationship tree.

[0131] Optionally, the first fault-related business nodes included in the second node relationship tree are business nodes that respond to the first read and write requests in the first node relationship tree, and the data flow between the first fault-related business nodes in the second node relationship tree is used to indicate the response order of the first fault-related business nodes to the first read and write requests.

[0132] Optionally, the first node relationship tree further indicates the data flow between the business nodes where IO occurs in the storage system. The data flow between the business nodes in the first node relationship tree is used to represent the calling relationship between the business nodes indicated by the storage system code script.

[0133] Optionally, the second node relationship tree is a candidate node relationship tree that best matches a historical fault node relationship tree in the historical fault database among multiple candidate node relationship trees in the first node relationship tree.

[0134] Optionally, the fault root cause node is determined based on the performance correlation between candidate fault root cause nodes in the second node relationship tree, and the candidate fault root cause node is a business node in a third node relationship tree in the second node relationship tree. The third node relationship tree is a node relationship tree with the highest-level fault business node in the second node relationship tree as the root node, and the fault business node is a business node in the second node relationship tree whose performance data is not within the target range.

[0135] Optionally, the fault root cause node is a candidate fault root cause node in the third node relationship tree, whose performance correlation with the service node of the previous level is greater than a preset value and whose performance correlation with the service node of the next level is less than the preset value.

[0136] Optionally, the display module 1101 is further configured to:

[0137] Based on the first node relationship tree, displaying a fourth node relationship tree of the storage system, the fourth node relationship tree including the service node related to the second fault in the service node where the IO occurs and the data flow between the service nodes related to the second fault;

[0138] The determining module 1102 is further configured to determine a fault root cause node of the second fault in the fourth node relationship tree based on performance data of each service node in the fourth node relationship tree.

[0139] Optionally, the device further comprises:

[0140] a receiving module, configured to receive a node relationship tree configuration instruction from a user, the node relationship tree configuration instruction being used to adjust at least one of the service nodes, performance data of the service nodes, and data flows between the service nodes in the second node relationship tree;

[0141] An adjustment module is configured to adjust the second node relationship tree based on the node relationship tree configuration instruction.

[0142] It should be noted that, in other embodiments, the steps that the above modules are responsible for implementing can be specified as needed, and the above modules can respectively implement different steps in the above storage system fault root cause locating method to realize all the functions of the above device. That is, the storage system fault root cause locating device provided in the above embodiment only uses the division of the above functional modules as an example to illustrate when implementing the storage system fault root cause locating method. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device provided in the above embodiment and the corresponding method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0143] The display module 1101 and the determination module 1102 can be implemented by software or hardware. For example, the implementation of the display module 1101 is described below using the display module 1101 as an example. Similarly, the implementation of the determination module 1102 can refer to the implementation of the display module 1101.

[0144] As an example of a software functional unit, the display module 1101 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the display module 1101 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0145] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0146] As an example of a hardware functional unit, the display module 1101 may include at least one computing device, such as a server. Alternatively, the display module 1101 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0147] The multiple computing devices included in the display module 1101 can be distributed in the same region or in different regions. The multiple computing devices included in the display module 1101 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the display module 1101 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0148] The present application also provides a computing device 1200 . Figure 12 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application, such as Figure 12 As shown, computing device 1200 includes a bus 1201, a processor 1202, a memory 1203, and a communication interface 1204. Processor 1202, memory 1203, and communication interface 1204 communicate with each other via bus 1201. Computing device 1200 can be a computing device or a terminal device. It should be understood that this application does not limit the number of processors and memories in computing device 1200.

[0149] The bus 1201 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 12The bus 1201 may include a path for transmitting information between various components of the computing device 1200 (eg, the memory 1203, the processor 1202, and the communication interface 1204).

[0150] The processor 1202 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0151] The memory 1203 may include a volatile memory, such as a random access memory (RAM). The memory 1203 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0152] Memory 1203 stores executable program code. Processor 1202 executes the executable program code to implement the functions of display module 1101 and determination module 1102, thereby implementing the method for locating the root cause of a storage system fault. Specifically, memory 1203 stores instructions for executing the method for locating the root cause of a storage system fault.

[0153] The communication interface 1204 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 1200 and other devices or a communication network.

[0154] An embodiment of the present application also provides a computing device cluster. Figure 13 is a schematic diagram of a computing device cluster provided in an embodiment of the present application, such as Figure 13 As shown, the computing device cluster includes at least one computing device 1200. The memory 1203 in one or more computing devices 1200 in the computing device cluster may store the same instructions for executing the storage system fault root cause location method.

[0155] In some possible implementations, the memory 1203 of one or more computing devices 1200 in the computing device cluster may also store partial instructions for executing the storage system fault root cause location method. In other words, the combination of one or more computing devices 1200 can collectively execute the instructions for executing the storage system fault root cause location method.

[0156] It should be noted that the memory 1203 in different computing devices 1200 in the computing device cluster can store different instructions, each for executing a portion of the functions of the storage system fault root cause location apparatus. In other words, the instructions stored in the memory 1203 in different computing devices 1200 can implement the functions of one or more of the aforementioned display module 1101 and determination module 1102.

[0157] It should be understood that Figure 13 The functionality of the computing device 1200 shown in FIG. 1 may also be performed by multiple computing devices 1200 .

[0158] In some possible implementations, one or more computing devices in a computing device cluster may be connected via a network, which may be a wide area network or a local area network. Figure 14 A possible implementation is shown. Figure 14 This is a schematic diagram of a possible implementation of a computing device cluster provided in an embodiment of the present application, such as Figure 14 As shown, two computing devices 1200 are connected via a network. Specifically, the connection to the network is achieved through a communication interface in each computing device.

[0159] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similarly referred to as Figure 13 or Figure 14 The difference is that the memory 1203 in one or more computing devices 1200 in the computing device cluster may store the same instructions for executing the storage system fault root cause locating method.

[0160] In some possible implementations, the memory 1203 of one or more computing devices 1200 in the computing device cluster may also store partial instructions for executing the storage system fault root cause location method. In other words, the combination of one or more computing devices 1200 can collectively execute the instructions for executing the storage system fault root cause location method.

[0161] It should be noted that the memory 1203 in different computing devices 1200 in the computing device cluster can store different instructions, each for executing a portion of the functions of the storage system fault root cause location apparatus. In other words, the instructions stored in the memory 1203 in different computing devices 1200 can implement the functions of one or more of the aforementioned display module 1101 and determination module 1102.

[0162] This application provides a computer program product. This computer program product may be software or a program product containing program code that can be run on a server, a terminal, or stored on any available medium. When this computer program product is run on at least one computing device, it causes the at least one computing device to execute the storage system fault root cause locating method provided in the above-described method embodiment.

[0163] An embodiment of the present application provides a computer-readable storage medium, which can be any available medium capable of being stored by a computing device, or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes program code, and when the program code is executed by at least one computing device, the at least one computing device executes the storage system fault root cause locating method provided in the above-described method embodiment.

[0164] It should be noted that the information involved in this application (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the business nodes and performance data involved in this application are all obtained with full authorization. In some embodiments, the embodiment of the present application provides a permission inquiry page, which is used to inquire whether the permission to obtain the above information is granted. In the permission inquiry page, an authorization consent control and an authorization rejection control are displayed. When a trigger operation for the authorization consent control is detected, the storage system fault root cause locating method provided by the embodiment of the present application is used to obtain the above information.

[0165] Those skilled in the art will appreciate that the various method steps and units described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the steps and components of each embodiment have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0166] Those skilled in the art will clearly understand that, for the sake of convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0167] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the unit is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or can be electrical, mechanical or other forms of connection.

[0168] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0169] In addition, each unit in each embodiment of the present application may be integrated into a processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The above-mentioned integrated units may be implemented in the form of hardware or software units.

[0170] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computing device (which can be a personal computer, a server, or a computing device, etc.) to execute all or part of the steps of the method in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0171] In this application, the terms "first", "second", etc. are used to distinguish between identical or similar items having substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is the quantity and execution order limited. It should also be understood that although the following description uses the terms first, second, etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of various examples, the first node relationship tree can be referred to as the second node relationship tree, and similarly, the second node relationship tree can be referred to as the first node relationship tree. Both the first node relationship tree and the second node relationship tree can be node relationship trees, and in some cases, can be separate and different node relationship trees.

[0172] The term "at least one" in this application means one or more, and the term "plurality" in this application means two or more. The terms "system" and "network" are often used interchangeably in this document.

[0173] It should also be understood that the term "if" may be interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting." Similarly, the phrase "if it is determined that..." or "if [stated condition or event] is detected" may be interpreted to mean "upon determining that..." or "in response to determining that..." or "upon detecting [stated condition or event]" or "in response to detecting [stated condition or event]," depending on the context.

[0174] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

[0175] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer program instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0176] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired or wireless method. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, or a magnetic tape), an optical medium (e.g., a digital video disc (DVD), or a semiconductor medium (e.g., a solid-state drive)), etc.

[0177] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0178] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for locating the root cause of a storage system failure, characterized in that: The method comprises: Displaying a first node relationship tree of a storage system, where the first node relationship tree indicates a service node in which IO occurs in the storage system; Based on the first node relationship tree, displaying a second node relationship tree of the storage system, where the second node relationship tree includes a service node related to the first fault among the service nodes where the IO occurs and data flows between the service nodes related to the first fault; Based on the performance data of each service node in the second node relationship tree, a fault root cause node of the first fault is determined in the second node relationship tree.

2. The method according to claim 1, characterized in that The first fault-related business nodes included in the second node relationship tree are the business nodes that respond to the first read and write requests in the first node relationship tree, and the data flow between the first fault-related business nodes in the second node relationship tree is used to indicate the response order of the first fault-related business nodes to the first read and write requests.

3. The method according to claim 1, characterized in that The first node relationship tree also indicates the data flow between the business nodes where IO occurs in the storage system. The data flow between the business nodes in the first node relationship tree is used to represent the calling relationship between the business nodes indicated by the storage system code script.

4. The method according to claim 3, characterized in that The second node relationship tree is a candidate node relationship tree that best matches a historical fault node relationship tree in a historical fault database among multiple candidate node relationship trees in the first node relationship tree.

5. The method according to any one of claims 1 to 4, characterized in that The fault root cause node is determined based on the performance correlation between the candidate fault root cause nodes in the second node relationship tree. The candidate fault root cause node is a business node in the third node relationship tree in the second node relationship tree. The third node relationship tree is a node relationship tree with the highest-level fault business node in the second node relationship tree as the root node. The fault business node is a business node in the second node relationship tree whose performance data is not within the target range.

6. The method according to claim 5, characterized in that The fault root cause node is a candidate fault root cause node in the third node relationship tree, whose performance correlation with the service node of the previous level is greater than a preset value and whose performance correlation with the service node of the next level is less than the preset value.

7. The method according to claim 1, characterized in that The method further comprises: Based on the first node relationship tree, displaying a fourth node relationship tree of the storage system, the fourth node relationship tree including data flows between a service node related to the second fault and a service node related to the second fault in the service node where the IO occurs; Based on the performance data of each service node in the fourth node relationship tree, a fault root cause node of the second fault is determined in the fourth node relationship tree.

8. The method according to any one of claims 1 to 7, characterized in that The method further comprises: receiving a node relationship tree configuration instruction from a user, wherein the node relationship tree configuration instruction is used to adjust at least one of a service node, performance data of the service node, and a data flow between service nodes in the second node relationship tree; Based on the node relationship tree configuration instruction, the second node relationship tree is adjusted.

9. A device for locating the root cause of a storage system failure, characterized in that: The device comprises: A display module, configured to display a first node relationship tree of the storage system, wherein the first node relationship tree indicates a service node in which IO occurs in the storage system; The display module is further configured to display a second node relationship tree of the storage system based on the first node relationship tree, wherein the second node relationship tree includes a service node related to the first fault among the service nodes where the IO occurs and data flows between the service nodes related to the first fault; A determination module is configured to determine a root cause node of the first fault in the second node relationship tree based on performance data of each service node in the second node relationship tree.

10. A computing device, characterized in that The computing device includes a processor and a memory, wherein the processor is configured to execute program code in the memory, so that the computing device executes the storage system fault root cause locating method according to any one of claims 1 to 8.

11. A computing device cluster, characterized in that: comprising at least one computing device, each of said computing devices comprising a processor and a memory; The processor of the at least one computing device is configured to execute program code stored in a memory of the at least one computing device, so that the computing device cluster executes the storage system fault root cause locating method according to any one of claims 1 to 8.

12. A storage medium, characterized in that: The storage medium stores at least one program code, and the at least one program code is read by a processor of a computing device to enable the computing device to execute the storage system fault root cause locating method according to any one of claims 1 to 8.

13. A storage medium, characterized in that: The storage medium stores at least one program code, and the at least one program code is read by a processor of a computing device in a computing device cluster to enable the computing device cluster to execute the storage system fault root cause locating method according to any one of claims 1 to 8.

Images