Method, device and equipment for locating root cause of failure of server cluster

Abstract

The present application provides a method, device and equipment for locating the root cause of a server cluster fault, and relates to the field of operation and maintenance technology. The method includes: obtaining the business link tracking data corresponding to the server cluster, and the business data of multiple servers in the server cluster; based on the business link tracking data, determining the status of each of the multiple servers, and the link node information of each business link, the status is normal or faulty, and the link node information is used to indicate the connection relationship between at least one server included in the corresponding business link; based on the business data and status of each of the multiple servers, determining the faulty business link from at least one business link; based on the link node information of the faulty business link, determining the faulty root cause server of the faulty business link. The solution of the present application can locate the root cause of the fault without checking the operation logs of each server one by one, which improves the efficiency of locating the root cause of the fault.

Description

Technical Field

[0001] The present application relates to the field of operation and maintenance technology, and in particular to a method, device and equipment for locating the root cause of a fault in a server cluster. Background Art

[0002] A server cluster is a cluster composed of multiple servers. Different servers in the server cluster can form corresponding business links to provide corresponding business services.

[0003] Currently, during the operation and maintenance of the business system of the server cluster, if a server in the server cluster fails, it will affect the business services of the corresponding business links. Therefore, it is necessary to locate the root cause of the failure in a timely manner and determine the location of the failure to ensure the normal operation of the business service.

[0004] Currently, when a server cluster fails, the operation and maintenance personnel need to check the server logs and find out the causes of the failure one by one. However, this method of locating the root cause of the failure is inefficient. Summary of the invention

[0005] The present application provides a method, device and equipment for locating the root cause of a fault in a server cluster, so as to solve the problem that the current method for locating the root cause of a fault is inefficient.

[0006] In a first aspect, the present application provides a method for locating the root cause of a fault in a server cluster, comprising:

[0007] Acquire business link tracking data corresponding to a server cluster and business data of multiple servers in the server cluster, wherein the server cluster includes at least one business link, and each business link includes at least one server;

[0008] Based on the service link tracking data, determine the status of each of the multiple servers and link node information of each service link, the status being normal or faulty, and the link node information being used to indicate a connection relationship between at least one server included in the corresponding service link;

[0009] Determining a faulty service link from the at least one service link based on service data and status of each of the plurality of servers;

[0010] Based on the link node information of the faulty service link, a fault root cause server of the faulty service link is determined.

[0011] In a possible implementation, determining the states of the multiple servers and the link node information of each service link based on the service link tracking data includes:

[0012] Slicing the service link tracking data to obtain a plurality of corresponding service link tracking files;

[0013] For each service link tracking file, the service link tracking file is processed to obtain status information of each server included in the target service link corresponding to the service link tracking file and link node information of the target service link, wherein the status information is used to indicate the status of the corresponding server;

[0014] The at least one service link includes target service links corresponding to the multiple service link tracking files, and the multiple servers include servers included in each of the target service links.

[0015] In a possible implementation manner, the service data includes a single service success rate indicator, a multi-service success rate indicator, and a service status indicator, and determining a faulty service link from the at least one service link based on the service data and status of each of the multiple servers includes:

[0016] For each server among the multiple servers, determining a service processing result of the server based on a single service success rate indicator, a multi-service success rate indicator, and a service status indicator of the server, the service processing result being a successful service processing or a failed service processing;

[0017] Among the multiple servers, determine a first server whose status is faulty and whose business processing result is business processing failure;

[0018] The service link including the service link of the first server among the at least one service link is determined as the faulty service link.

[0019] In a possible implementation, the link node information includes a node identifier of a server and a parent node identifier of the server, and determining the root cause server of the faulty service link based on the link node information of the faulty service link includes:

[0020] Based on the node identifier and the parent node identifier of each server in the faulty service link, construct a fault tree corresponding to the faulty service link;

[0021] Based on the fault tree, a fault root cause server of the faulty service link is determined.

[0022] In a possible implementation manner, determining the root cause server of the faulty service link based on the fault tree includes:

[0023] Determine the number of second servers in a fault state included in the fault tree;

[0024] In a case where the number of the second server is one, determining the second server as a fault root cause server of the faulty service link;

[0025] When the number of the second servers is at least two, the fault root cause server of the faulty service link is determined based on the levels of the at least two second servers in the fault tree.

[0026] In a possible implementation manner, determining the root cause server of the fault service link based on the levels of at least two second servers in the fault tree includes:

[0027] Determine, from the at least two second servers, a third server belonging to the highest level in the fault tree;

[0028] In a case where the number of the third server is one, determining the third server as a fault root cause server of the faulty service link;

[0029] When the number of the third servers is at least two, based on the time information in the link node information of each of the at least two third servers, determine the fourth server with the earliest time information from the at least two third servers; and determine the fourth server as the root cause server of the faulty service link.

[0030] In a possible implementation, the method further includes:

[0031] Inputting the fault information of the root cause server into the operation and maintenance big model to generate a business solution strategy corresponding to the root cause server;

[0032] Based on the business solution strategy, processing the fault information of the root cause server;

[0033] The operation and maintenance big model is trained based on sample fault information of multiple fault root cause server samples and corresponding business solution strategy labels.

[0034] In a second aspect, the present application provides a device for locating the root cause of a fault in a server cluster, comprising:

[0035] An acquisition module, used to acquire business link tracking data corresponding to a server cluster and business data of multiple servers in the server cluster, wherein the server cluster includes at least one business link, and each business link includes at least one server;

[0036] A determination module, configured to determine, based on the service link tracking data, the status of each of the plurality of servers and link node information of each service link, wherein the status is normal or faulty, and the link node information is used to indicate a connection relationship between at least one server included in the corresponding service link;

[0037] A first processing module, configured to determine a faulty service link from the at least one service link based on service data and status of each of the plurality of servers;

[0038] The second processing module is used to determine the fault root cause server of the faulty service link based on the link node information of the faulty service link.

[0039] In a possible implementation manner, the determining module is specifically configured to:

[0040] Slicing the service link tracking data to obtain a plurality of corresponding service link tracking files;

[0041] For each service link tracking file, the service link tracking file is processed to obtain status information of each server included in the target service link corresponding to the service link tracking file and link node information of the target service link, wherein the status information is used to indicate the status of the corresponding server;

[0042] The at least one service link includes target service links corresponding to the multiple service link tracking files, and the multiple servers include servers included in each of the target service links.

[0043] In a possible implementation manner, the service data includes a single service success rate indicator, a multi-service success rate indicator, and a service status indicator, and the first processing module is specifically used to:

[0044] For each server among the multiple servers, determining a service processing result of the server based on a single service success rate indicator, a multi-service success rate indicator, and a service status indicator of the server, the service processing result being a successful service processing or a failed service processing;

[0045] Among the multiple servers, determine a first server whose status is faulty and whose business processing result is business processing failure;

[0046] The service link including the service link of the first server among the at least one service link is determined as the faulty service link.

[0047] In a possible implementation manner, the link node information includes a node identifier of the server and a parent node identifier of the server, and the second processing module is specifically configured to:

[0048] Based on the node identifier and the parent node identifier of each server in the faulty service link, construct a fault tree corresponding to the faulty service link;

[0049] Based on the fault tree, a fault root cause server of the faulty service link is determined.

[0050] In a possible implementation manner, the second processing module is specifically configured to:

[0051] Determine the number of second servers whose status information included in the fault tree is faulty;

[0052] In a case where the number of the second server is one, determining the second server as a fault root cause server of the faulty service link;

[0053] When the number of the second servers is at least two, the fault root cause server of the faulty service link is determined based on the levels of the at least two second servers in the fault tree.

[0054] In a possible implementation manner, the second processing module is specifically configured to:

[0055] Determine, from the at least two second servers, a third server belonging to the highest level in the fault tree;

[0056] In a case where the number of the third server is one, determining the third server as a fault root cause server of the faulty service link;

[0057] When the number of the third servers is at least two, based on the time information in the link node information of each of the at least two third servers, determine the fourth server with the earliest time information from the at least two third servers; and determine the fourth server as the root cause server of the faulty service link.

[0058] In a possible implementation manner, the second processing module is further configured to:

[0059] Inputting the fault information of the root cause server into the operation and maintenance big model to generate a business solution strategy corresponding to the root cause server;

[0060] Based on the business solution strategy, processing the fault information of the root cause server;

[0061] The operation and maintenance big model is trained based on sample fault information of multiple fault root cause server samples and corresponding business solution strategy labels.

[0062] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for locating the root cause of a fault in a server cluster as described in any one of the first aspects is implemented.

[0063] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for locating the root cause of a fault in a server cluster as described in any one of the first aspects is implemented.

[0064] The method, device and equipment for locating the root cause of a fault in a server cluster provided by the present application first obtain the service link tracking data corresponding to the server cluster and the service data of multiple servers in the server cluster. The server cluster includes at least one service link, and each service link includes at least one server. Then, based on the service link tracking data, the status of each of the multiple servers and the link node information of each service link are determined. The status of the server is normal or faulty. The link node information is used to indicate the connection relationship between at least one server included in the corresponding service link. Based on the service data and status of each of the multiple servers, the faulty service link can be determined from at least one service link, and the faulty service link can be determined based on the link node information of the faulty service link. The faulty service link includes at least one server. By determining the faulty service link through service data and service link tracking data, the root cause of the fault can be associated with the service logic, thereby improving the accuracy of locating the root cause of the fault. After determining the faulty service link, the server of the faulty service link can be determined based on the link node information of the faulty service link. The scheme of the present application can locate the root cause of the fault without checking the operation logs of each server one by one, thereby improving the efficiency of locating the root cause of the fault. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0066] Figure 1 A schematic diagram of an applicable application scenario provided for an embodiment of the present application;

[0067] Figure 2 A flowchart of a method for locating the root cause of a server cluster failure provided in an embodiment of the present application;

[0068] Figure 3 A schematic diagram of the process of locating the root cause of a fault provided in an embodiment of the present application;

[0069] Figure 4 A flow chart of a server for determining the root cause of a faulty service link provided in an embodiment of the present application;

[0070] Figure 5 A schematic diagram of a fault tree provided in an embodiment of the present application;

[0071] Figure 6 A schematic diagram of the structure of a device for locating the root cause of a fault in a server cluster provided in an embodiment of the present application;

[0072] Figure 7 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0074] Figure 1 A schematic diagram of an applicable application scenario provided in an embodiment of the present application, such as Figure 1 As shown, the server cluster 11 may include multiple servers. Figure 1 The example in FIG. 1 shows that the server cluster 11 includes a server 111, a server 112, a server 113, a server 114, a server 115, and a server 116. It is understandable that in Figure 1 The number of servers included in the server cluster 11 is only an example and does not constitute a limitation on the number of servers.

[0075] The servers in the server cluster 11 may form different service links, and different service links may provide the same or different service services. Figure 1 In the example, servers 111, 112, 113 and 114 constitute a service link A, servers 114, 115 and 116 constitute a service link B, and service link A and service link B are used to provide corresponding service services. After multiple servers constitute a service link, the multiple servers constituting the service link need to work together so that the service link provides corresponding service services.

[0076] During the operation and maintenance of the server cluster 11, some servers may fail, thereby affecting the business services of the corresponding business links. Therefore, in the event of a failure, it is necessary to locate the location of the failure in a timely manner to ensure the normal operation of the business services.

[0077] However, the server cluster 11 is usually large in scale, and the traditional method of checking the operation logs of each server to locate the root cause of the fault is very inefficient. Based on this, an embodiment of the present application provides a method for locating the root cause of a server cluster to improve the efficiency of locating the root cause of the fault. The solution of the embodiment of the present application will be introduced below in conjunction with the accompanying drawings.

[0078] Figure 2 A flowchart of a method for locating the root cause of a server cluster failure provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the method includes:

[0079] S21, obtaining business link tracking data corresponding to the server cluster and business data of multiple servers in the server cluster, the server cluster includes at least one business link, and each business link includes at least one server.

[0080] The execution subject of each embodiment of the present application may be, for example, a server, a processor, a microprocessor, etc., or a device that integrates a server, a processor, or a microprocessor, such as a terminal device, a client, etc. As long as it is a device with computing and processing capabilities, it can be used as the execution subject of each embodiment of the present application. Figure 1 In the embodiment, the management device 10 can be used as an execution subject to implement the root cause location of the fault of the server cluster 11. In the following embodiments, for the convenience of description, the execution subject is taken as an example of the management device.

[0081] The server cluster includes at least one service link. For each service link, the service link includes at least one server. During the operation of the server cluster, the corresponding service link tracking data will be continuously generated. The service link tracking data reflects the relevant information in each service link included in the server cluster, for example, it may include the link node information of each server, the service link identifier, etc. For any service link, the service link tracking data corresponding to the server cluster may also include the service call relationship between the servers in the service link.

[0082] The business data of the server is data related to the business processed by the server, and may include information such as the business processed by the server, the business success rate, the business status, etc. Each server included in the server cluster has corresponding business data.

[0083] S22, based on the business link tracking data, determine the status of each of the multiple servers and the link node information of each business link, the status is normal or faulty, and the link node information is used to indicate the connection relationship between at least one server included in the corresponding business link.

[0084] The service link tracking data includes the status information of each of the multiple servers. Through the status information of each of the multiple servers, the status of each of the multiple servers can be determined.

[0085] The business link tracking data records the link node information of each server in chronological order. The link node information can be used to indicate the business link to which the corresponding server belongs, its position in the business link, the connection relationship between it and other servers in the business link, etc.

[0086] S23: Determine a faulty service link from at least one service link based on service data and status of each of the multiple servers.

[0087] In a server cluster, multiple business links are involved. Figure 1 For example, server cluster 11 includes service link A and service link B. After acquiring service data and service link tracking data of each server, the service link tracking data can be mined for root causes based on the multi-fault engine, and the faulty service link can be determined in combination with the service data of each server.

[0088] The faulty service link is a service link in the service link corresponding to the server cluster and having a fault. The number of the faulty service links may be one or more, which is not limited in the embodiment of the present application. For any faulty service link, the faulty service link includes at least one server.

[0089] For each server, based on the business data of the server, the business processing result of the server can be determined, and the business processing result is business processing success or business processing failure. Then, from the multiple servers included in the server cluster, the server whose business processing failed is determined, and from the multiple servers included in the server cluster, the server whose status is faulty is determined.

[0090] Furthermore, the service link to which the server whose service processing fails belongs can be determined as a faulty service link; or, the service link to which the server whose status is faulty belongs can be determined as a faulty service link; or, the service link to which the server whose service processing fails and whose status is faulty belongs can be determined as a faulty service link, and so on. The embodiments of the present application are not limited to this.

[0091] S24: Determine a server that is a root cause of the fault of the faulty service link based on the link node information of the faulty service link.

[0092] After the faulty service link is determined, since the faulty service link includes at least one server and the at least one server includes a faulty server, it is necessary to determine the faulty server in the faulty service link, thereby determining the corresponding fault root cause server.

[0093] It should be noted that the root cause server is a server with a fault in the faulty service link, but not all servers with a fault are root cause servers. Usually, the first faulty server encountered in the process of processing services on the faulty service link is the root cause of the fault. Therefore, the first faulty server in the faulty service link can be determined based on the link node information of the faulty service link and determined as the corresponding root cause server.

[0094] The method for locating the root cause of a fault in a server cluster provided in an embodiment of the present application first obtains the service link tracking data corresponding to the server cluster and the service data of multiple servers in the server cluster. The server cluster includes at least one service link, and each service link includes at least one server. Then, based on the service link tracking data, the status of each of the multiple servers and the link node information of each service link are determined. The status of the server is normal or faulty. The link node information is used to indicate the connection relationship between at least one server included in the corresponding service link. Based on the service data and status of each of the multiple servers, the faulty service link can be determined from at least one service link, and the faulty service link can be determined based on the link node information of the faulty service link. By determining the faulty service link through service data and service link tracking data, the root cause of the fault can be associated with the service logic, thereby improving the accuracy of locating the root cause of the fault. After determining the faulty service link, the server of the faulty service link can be determined based on the link node information of the faulty service link. The scheme of the embodiment of the present application can locate the root cause of the fault without checking the operation logs of each server one by one, thereby improving the efficiency of locating the root cause of the fault.

[0095] Based on any of the above embodiments, the scheme of the embodiments of the present application is introduced below in conjunction with the drawings.

[0096] Figure 3 A schematic diagram of the process of locating the root cause of a fault provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, there are four processes in total.

[0097] The first process is to judge the outliers based on the business data of multiple servers. The outlier judgment is to determine the server where the business processing fails among multiple servers based on the business data.

[0098] The second process is to process the business link tracking data through the first multi-fault engine to determine the server in a faulty state, and combine the results of the abnormal point judgment in the first process to comprehensively determine the faulty business link.

[0099] The third process is to generate a fault tree of the fault service link through the second multi-fault engine.

[0100] The fourth process is to analyze the root cause of the faulty service link based on the fault tree, generate corresponding service solution strategies, and handle the corresponding faults.

[0101] The above four processes will be introduced respectively in conjunction with specific embodiments below.

[0102] After obtaining the service data and service link tracking data of each server, the faulty service link can be determined.

[0103] Specifically, for the business link tracking data, since the business link tracking data is usually large, directly processing the business link tracking data has high requirements on the management device. Therefore, in the embodiment of the present application, the business link tracking data is first segmented to obtain a plurality of corresponding business link tracking files, and then the plurality of business link tracking files are processed in turn.

[0104] The business link tracking data records the link node information of each server in chronological order, so the business link tracking data can be segmented according to the time information to obtain multiple corresponding business link tracking files. For example, a fixed time interval can be set in advance, and the business link tracking data can be segmented based on the fixed time interval. The multiple business link tracking files obtained by segmentation also include the link node information of the corresponding server in the corresponding time period.

[0105] For each service link tracking file, the management device performs the following operations:

[0106] The service link tracking file is processed to obtain status information of each server included in the target service link corresponding to the service link tracking file, and link node information of the target service link, wherein the status information is used to indicate the status of the corresponding server, and the status is faulty or normal. Among them, at least one service link includes target service links corresponding to each of the multiple service link tracking files, and the multiple servers include servers included in each of the target service links.

[0107] The business data of a server may include a single-business success rate indicator, a multi-business success rate indicator, and a business status indicator. For any server, based on the single-business success rate indicator, multi-business success rate indicator, and business status indicator of the server, it can be determined whether the server has an abnormality, that is, whether the server has a business processing failure.

[0108] Specifically, for each server in the server cluster, the business data of the server is obtained, including the single business success rate index, multi-business success rate index and business status index of the server. For each server in the multiple servers, based on the single business success rate index, multi-business success rate index and business status index of the server, the business processing result of the server is determined, and the business processing result is a successful business processing, or a business processing failure. For example, it can be determined whether each indicator of the server (i.e., the single business success rate index, the multi-business success rate index and the business status index) is greater than or equal to a preset threshold. In the case where each indicator of the server is greater than or equal to the preset threshold, it is determined that there is no business processing failure on the server. In the case where there is at least one indicator of the server that is less than the preset threshold, it is determined that there is a business processing failure on the server. For a server with a business processing failure, the node identifier and corresponding time information of the server will be recorded.

[0109] Among the multiple servers, a first server whose status is faulty and whose service processing fails is determined, and a service link including the first server in at least one service link is determined as a faulty service link.

[0110] Specifically, for each service link tracking file, determine whether the status of each of the corresponding multiple servers is a fault, determine a server in a faulty status among the corresponding multiple servers, and determine a server in which service processing fails among the servers in a faulty status as the first server. If at least one of the single service success rate index, the multi-service success rate index, and the service status index of the first server is less than a preset threshold, it is determined that the service processing of the first server has failed.

[0111] After the first server is determined, the service link including the first server may be determined as a faulty service link among the multiple service links corresponding to the service cluster.

[0112] In a possible implementation, multiple service link tracking files may be processed based on the first multi-fault engine to determine the faulty service link.

[0113] Specifically, first determine the number n of multiple business link tracking files, where n is a positive integer. Then determine the number m of threads supported by the first multi-fault engine, where m is a positive integer. Initialize the first multi-fault engine, and initialize the number of threads used to process business link tracking files to min[m,n], where min[m,n] represents the smaller value of m and n, and each initialized thread of the first multi-fault engine is used to process a business link tracking file.

[0114] If m is greater than n, there is no need to initialize all threads of the first multi-fault engine, and only n threads need to be initialized for processing business link tracking files. If m is less than or equal to n, all threads of the first multi-fault engine need to be initialized. When m is less than n, the first multi-fault engine cannot complete the processing of n business link tracking files at one time, so it is necessary to continue processing the remaining business link tracking files after some threads complete the processing flow.

[0115] The first multi-fault engine needs to search for a first server in a faulty state and with at least one indicator less than a preset threshold among multiple servers, and record the identifier of the service link to which the first server belongs, and the service link to which the first server belongs is a faulty service link.

[0116] In some embodiments, there may be a situation where the identifiers of the service links to which multiple first servers belong are the same, so the identifiers of the service links to which the first servers belong may be deduplicated to obtain a finally determined faulty service link.

[0117] In the above embodiment, the implementation method of determining the faulty service link is introduced. Figure 4 This section describes how to determine the root cause of a faulty service link.

[0118] Figure 4 A flowchart of determining the root cause of a faulty service link provided by an embodiment of the present application is shown in FIG. Figure 4 As shown, including:

[0119] S41, constructing a fault tree corresponding to the faulty service link based on the node identifier and the parent node identifier of each server in the faulty service link.

[0120] The link node information of the faulty service link includes the node identifier of the server and the parent node identifier of the server, wherein the node identifier of the server is used to indicate the server, and the parent node identifier of the server is used to indicate the parent node of the server in the faulty service link. Therefore, for at least one server included in the faulty service link, a fault tree corresponding to the faulty service link can be constructed based on the node identifier and the parent node identifier of each of the at least one server.

[0121] Specifically, the server indicated by the parent node identifier and the server indicated by the node identifier of each server under the same faulty service link may be connected in sequence, so as to obtain a fault tree corresponding to the faulty service link.

[0122] Figure 5 A schematic diagram of a fault tree provided in an embodiment of the present application is shown as follows: Figure 5As shown, a fault tree corresponding to a faulty service link is illustrated, which is visualized using the graphviz dependency package (a visualization dependency package of Python). In the fault tree, multiple servers are included, and the hierarchical relationship between the multiple servers and the status information of the multiple servers are illustrated. The status information is used to indicate whether the status of the corresponding server is faulty or normal. Figure 5 In each node, the server's ID and status are illustrated, and the arrows represent the hierarchical relationship between the servers, where server 1 is the parent node of server 1.1, server 1.2, server 1.3, and server 1.4, server 1.1 is the parent node of server 2.1, server 2.2, server 2.3, server 2.4, server 2.5, and server 2.6, and server 2.5 is the parent node of server 2.7. The status of server 2.5 is faulty, and the status of other servers is normal.

[0123] S42, based on the fault tree, determine the root cause server of the faulty service link.

[0124] First, determine the number of second servers in the fault tree that are in a fault state. If the number of second servers is one, determine the second server as the root cause server of the fault service link. Figure 5 In the example, there is only one second server in a faulty state, so the second server is the root cause server of the faulty service link.

[0125] When the number of the second servers is at least two, the fault root cause server of the faulty service link is determined based on the levels of the at least two second servers in the fault tree.

[0126] Specifically, in the case where the number of second servers is at least two, first determine the third server with the highest level in the fault tree from the at least two second servers, and determine the root cause server of the faulty service link based on the number of third servers.

[0127] The level to which a server belongs in the fault tree can be determined based on the parent node of each server. For example, the link node information of the faulty service link includes the node identifier of the server and the parent node identifier of the server. Based on the parent node identifier, the parent node server of the server can be determined, and the level of the parent node server is higher than that of the server. If there are multiple servers with the same parent node server, the levels of these multiple servers are the same, and they are all servers at the next level of the parent node server.

[0128] by Figure 5For example, server 1 is the root node server, with the highest level in the fault tree, recorded as the first level. Server 1 is the parent node server of server 1.1, server 1.2, server 1.3 and server 1.4, then server 1.1, server 1.2, server 1.3 and server 1.4 have the same level, and are all servers at the next level of server 1, recorded as the second level. Server 1.1 is the parent node server of server 2.1, server 2.2, server 2.3, server 2.4, server 2.5 and server 2.6, then server 2.1, server 2.2, server 2.3, server 2.4, server 2.5 and server 2.6 have the same level, and are all servers at the next level of server 1.1, recorded as the third level. Server 2.5 is the parent node server of server 2.7, then server 2.5 is the server at the next level of server 2.7, recorded as the fourth level.

[0129] For at least two second servers, the level to which each second server belongs in the fault tree can be determined first, and then the third server with the highest level in the fault tree can be determined. Figure 5 For example, if the status of server 1.2 and server 2.5 are both faulty and the service processing fails, then server 1.2 and server 2.5 are both second servers. The level of server 1.2 in the fault tree is the second level, and the level of server 2.5 in the fault tree is the third level. Then the third server with the highest level in the fault tree is server 1.2.

[0130] When the number of the third server is one, the third server is determined as the root cause server of the faulty service link. Figure 5 In the example, taking the third server as server 2.5, server 2.5 is determined as the root cause server of the faulty service link.

[0131] When the number of third servers is at least two, based on the time information in the link node information of each of the at least two third servers, the fourth server with the earliest time information is determined from the at least two third servers, and the fourth server is determined as the root cause server of the faulty service link.

[0132] In an embodiment of the present application, the server time information refers to the time information of calling the server to perform related business processing in the faulty business link. The earlier the server time information, the earlier the faulty business link schedules the server to process the related business. The later the server time information, the later the faulty business link schedules the server to process the related business.

[0133] Figures 4 to 5The embodiment is a process of processing a single faulty service link and determining the corresponding root cause server of the fault. In practice, there may be multiple faulty service links, and these multiple faulty service links can be processed by a second multi-fault engine, and the second multi-fault engine is in a cascade relationship with the first multi-fault engine. Suppose the number of multiple faulty service links is t, where t is a positive integer, and the number of threads supported by the second multi-fault engine is k, where k is a positive integer. Then the second multi-fault engine can be initialized, and the number of threads initialized for processing the faulty service link is min[k,t], where min[k,t] represents the smaller value of k and t, and each initialized thread of the second multi-fault engine is used to process a faulty service link.

[0134] If k is greater than t, there is no need to initialize all threads of the second multi-fault engine, and only t threads need to be initialized to process the faulty business link. If k is less than or equal to t, all threads of the second multi-fault engine need to be initialized. When k is less than t, the second multi-fault engine cannot complete the processing of t business link tracking files at one time, so it is necessary to continue processing the remaining faulty business links after some threads complete the processing flow.

[0135] After determining the root cause server, the management device inputs the fault information of the root cause server into the operation and maintenance model to generate a business solution strategy corresponding to the root cause server. The fault information of the root cause server may include, for example, the identification of the root cause server, the next server node of the root cause server, the fault code information of the root cause server, application information, etc., which is not limited in the embodiment of the present application. Based on the business solution strategy, the fault information of the root cause server can be processed.

[0136] Among them, the operation and maintenance big model is trained based on the sample fault information of multiple fault root cause server samples and the corresponding business solution strategy labels. Initially, the sample fault information of multiple fault root cause server samples is input into the operation and maintenance big model, the operation and maintenance big model processes the sample fault information, inputs the corresponding business solution strategy, and then determines the corresponding loss value based on the business solution strategy and the corresponding business solution strategy label output by the operation and maintenance big model, and adjusts the parameters of the operation and maintenance big model based on the loss value, thereby completing a round of training process of the operation and maintenance big model. After multiple rounds of training process, after reaching the termination condition of model training, a trained operation and maintenance big model can be generated. The trained operation and maintenance big model has the ability to generate business solution strategies. Therefore, the fault information of the fault root cause server is input into the operation and maintenance big model, and the business solution strategy corresponding to the fault root cause server can be generated to solve the fault of the fault root cause server and ensure that the corresponding fault business link is restored to normal.

[0137] In summary, the solution of the embodiment of the present application determines the faulty business link through business data and business link tracking data, and can associate the root cause of the fault with the business logic, thereby improving the accuracy of locating the root cause of the fault. The root cause of the fault is located by adopting a multi-fault engine for query and real-time generation of a fault tree. The business solution strategy is generated in real time in combination with the operation and maintenance big model. The root cause of the fault can be located and processed without checking the operation logs of each server one by one, thereby improving the efficiency of locating and processing the root cause of the fault.

[0138] Figure 6 A schematic diagram of the structure of a device for locating the root cause of a server cluster failure provided in an embodiment of the present application, such as Figure 6 As shown, the device comprises:

[0139] An acquisition module 61 is used to acquire business link tracking data corresponding to a server cluster and business data of multiple servers in the server cluster, wherein the server cluster includes at least one business link, and each business link includes at least one server;

[0140] A determination module 62, configured to determine the status of each of the plurality of servers and link node information of each service link based on the service link tracking data, wherein the status is normal or faulty, and the link node information is used to indicate a connection relationship between at least one server included in the corresponding service link;

[0141] A first processing module 63, configured to determine a faulty service link from the at least one service link based on the service data and status of each of the plurality of servers;

[0142] The second processing module 64 is configured to determine a fault root cause server of the faulty service link based on the link node information of the faulty service link.

[0143] In a possible implementation manner, the determining module 62 is specifically configured to:

[0144] Slicing the service link tracking data to obtain a plurality of corresponding service link tracking files;

[0145] For each service link tracking file, the service link tracking file is processed to obtain status information of each server included in the target service link corresponding to the service link tracking file and link node information of the target service link, wherein the status information is used to indicate the status of the corresponding server;

[0146] The at least one service link includes target service links corresponding to the multiple service link tracking files, and the multiple servers include servers included in each of the target service links.

[0147] In a possible implementation manner, the service data includes a single service success rate indicator, a multi-service success rate indicator, and a service status indicator, and the first processing module 63 is specifically used to:

[0148] For each server among the multiple servers, determining a service processing result of the server based on a single service success rate indicator, a multi-service success rate indicator, and a service status indicator of the server, the service processing result being a successful service processing or a failed service processing;

[0149] Among the multiple servers, determine a first server whose status is faulty and whose business processing result is business processing failure;

[0150] The service link including the service link of the first server among the at least one service link is determined as the faulty service link.

[0151] In a possible implementation manner, the link node information includes a node identifier of the server and a parent node identifier of the server, and the second processing module 64 is specifically configured to:

[0152] Based on the node identifier and the parent node identifier of each server in the faulty service link, construct a fault tree corresponding to the faulty service link;

[0153] Based on the fault tree, a fault root cause server of the faulty service link is determined.

[0154] In a possible implementation manner, the second processing module 64 is specifically configured to:

[0155] Determine the number of second servers whose status information included in the fault tree is faulty;

[0156] In a case where the number of the second server is one, determining the second server as a fault root cause server of the faulty service link;

[0157] When the number of the second servers is at least two, the fault root cause server of the faulty service link is determined based on the levels of the at least two second servers in the fault tree.

[0158] In a possible implementation manner, the second processing module 64 is specifically configured to:

[0159] Determine, from the at least two second servers, a third server belonging to the highest level in the fault tree;

[0160] In a case where the number of the third server is one, determining the third server as a fault root cause server of the faulty service link;

[0161] When the number of the third servers is at least two, based on the time information in the link node information of each of the at least two third servers, determine the fourth server with the earliest time information from the at least two third servers; and determine the fourth server as the root cause server of the faulty service link.

[0162] In a possible implementation manner, the second processing module 64 is further configured to:

[0163] Inputting the fault information of the root cause server into the operation and maintenance big model to generate a business solution strategy corresponding to the root cause server;

[0164] Based on the business solution strategy, processing the fault information of the root cause server;

[0165] The operation and maintenance big model is trained based on sample fault information of multiple fault root cause server samples and corresponding business solution strategy labels.

[0166] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the method for locating the root cause of a fault in a server cluster, the method comprising: obtaining the service link tracking data corresponding to the server cluster and the service data of multiple servers in the server cluster, the server cluster comprising at least one service link, each service link comprising at least one server; based on the service link tracking data, determining the status of each of the multiple servers and the link node information of each service link, the status being normal or faulty, the link node information being used to indicate the connection relationship between at least one server included in the corresponding service link; based on the service data and status of each of the multiple servers, determining the faulty service link from the at least one service link; based on the link node information of the faulty service link, determining the faulty service link of the faulty service link.

[0167] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0168] On the other hand, the present application also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the method for locating the root cause of the fault of a server cluster provided in the above-mentioned embodiments, the method including: obtaining business link tracking data corresponding to the server cluster, and business data of multiple servers in the server cluster, the server cluster including at least one business link, each business link including at least one server; based on the business link tracking data, determining the status of each of the multiple servers and the link node information of each business link, the status is normal or faulty, and the link node information is used to indicate the connection relationship between at least one server included in the corresponding business link; based on the business data and status of each of the multiple servers, determining a faulty business link from the at least one business link; based on the link node information of the faulty business link, determining the root cause server of the faulty business link.

[0169] On the other hand, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the method for locating the root cause of a server cluster fault provided in the above-mentioned embodiments, the method comprising: obtaining business link tracking data corresponding to the server cluster, and business data of multiple servers in the server cluster, the server cluster comprising at least one business link, each business link comprising at least one server; based on the business link tracking data, determining the respective states of the multiple servers, and link node information of each business link, the state being normal or faulty, and the link node information being used to indicate the connection relationship between at least one server included in the corresponding business link; based on the respective business data and states of the multiple servers, determining a faulty business link from the at least one business link; based on the link node information of the faulty business link, determining the root cause server of the faulty business link.

[0170] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0171] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for locating the root cause of a server cluster failure, characterized in that: include: Acquire business link tracking data corresponding to a server cluster and business data of multiple servers in the server cluster, wherein the server cluster includes at least one business link, and each business link includes at least one server; Based on the service link tracking data, determine the status of each of the multiple servers and link node information of each service link, the status being normal or faulty, and the link node information being used to indicate a connection relationship between at least one server included in the corresponding service link; Determining a faulty service link from the at least one service link based on service data and status of each of the plurality of servers; Determine a fault root cause server of the faulty service link based on the link node information of the faulty service link; The service data includes a single service success rate indicator, a multi-service success rate indicator, and a service status indicator, and determining a faulty service link from the at least one service link based on the service data and status of each of the multiple servers includes: For each server among the multiple servers, determining a service processing result of the server based on a single service success rate indicator, a multi-service success rate indicator, and a service status indicator of the server, the service processing result being a successful service processing or a failed service processing; Among the multiple servers, determine a first server whose status is faulty and whose business processing result is business processing failure; The service link including the service link of the first server among the at least one service link is determined as the faulty service link.

2. The method according to claim 1, characterized in that The determining, based on the service link tracking data, the states of the plurality of servers and the link node information of each service link includes: Slicing the service link tracking data to obtain a plurality of corresponding service link tracking files; For each service link tracking file, the service link tracking file is processed to obtain status information of each server included in the target service link corresponding to the service link tracking file and link node information of the target service link, wherein the status information is used to indicate the status of the corresponding server; The at least one service link includes target service links corresponding to the multiple service link tracking files, and the multiple servers include servers included in each of the target service links.

3. The method according to claim 1 or 2, characterized in that: The link node information includes a node identifier of a server and a parent node identifier of the server, and determining the root cause server of the faulty service link based on the link node information of the faulty service link includes: Based on the node identifier and the parent node identifier of each server in the faulty service link, construct a fault tree corresponding to the faulty service link; Based on the fault tree, a fault root cause server of the faulty service link is determined.

4. The method according to claim 3, characterized in that The step of determining the root cause server of the fault service link based on the fault tree includes: Determine the number of second servers in a fault state included in the fault tree; In a case where the number of the second server is one, determining the second server as a fault root cause server of the faulty service link; When the number of the second servers is at least two, the fault root cause server of the faulty service link is determined based on the levels of the at least two second servers in the fault tree.

5. The method according to claim 4, characterized in that The determining the root cause server of the fault service link based on the levels of at least two second servers in the fault tree includes: Determine, from the at least two second servers, a third server belonging to the highest level in the fault tree; In a case where the number of the third server is one, determining the third server as a fault root cause server of the faulty service link; When the number of the third servers is at least two, based on the time information in the link node information of each of the at least two third servers, determine the fourth server with the earliest time information from the at least two third servers; and determine the fourth server as the root cause server of the faulty service link.

6. The method according to claim 1 or 2, characterized in that: The method further comprises: Inputting the fault information of the root cause server into the operation and maintenance big model to generate a business solution strategy corresponding to the root cause server; Based on the business solution strategy, processing the fault information of the root cause server; The operation and maintenance big model is trained based on sample fault information of multiple fault root cause server samples and corresponding business solution strategy labels.

7. A device for locating the root cause of a server cluster failure, characterized in that: include: An acquisition module, used to acquire business link tracking data corresponding to a server cluster and business data of multiple servers in the server cluster, wherein the server cluster includes at least one business link, and each business link includes at least one server; A determination module, configured to determine, based on the service link tracking data, the status of each of the plurality of servers and link node information of each service link, wherein the status is normal or faulty, and the link node information is used to indicate a connection relationship between at least one server included in the corresponding service link; A first processing module, configured to determine a faulty service link from the at least one service link based on service data and status of each of the plurality of servers; A second processing module, configured to determine a fault root cause server of the faulty service link based on the link node information of the faulty service link; The service data includes a single service success rate indicator, a multi-service success rate indicator and a service status indicator, and the first processing module is specifically used for: For each server among the multiple servers, determining a service processing result of the server based on a single service success rate indicator, a multi-service success rate indicator, and a service status indicator of the server, the service processing result being a successful service processing or a failed service processing; Among the multiple servers, determine a first server whose status is faulty and whose business processing result is business processing failure; The service link including the service link of the first server among the at least one service link is determined as the faulty service link.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for locating the root cause of a fault in a server cluster as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for locating the root cause of a fault in a server cluster as claimed in any one of claims 1 to 6 is implemented.

Images