Method, device, electronic device and storage medium for locating root cause of business failure
By determining the abnormal moment and call chain information of the target business in the business system, combining the device topology relationship and association rules, using undirected graphs and maximum spanning tree algorithms, the fast and accurate root cause positioning of business failures is achieved, and the existing solutions are solved Positioning difficult problems in technology has improved operation and maintenance efficiency and system monitoring capabilities.
Patent Information
- Application Number
- CN202110996024.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-08-27
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2041-08-27
AI Technical Summary
The existing technology cannot quickly and accurately locate the core root cause of business system failures, resulting in high operation and maintenance pressure and it is difficult to achieve rapid fault positioning.
By determining the target indicator abnormal moment of the target service, determining the root cause node based on the call chain information of the abnormal moment, and determining the root cause alarm based on the device topology relationship and preset association rules, using undirected graph and maximum spanning tree algorithm for association analysis , achieve fast and accurate root cause positioning.
It realizes rapid and accurate positioning of business system failures, reduces operation and maintenance complexity, integrates operation and maintenance guarantees, avoids positioning difficulties caused by layered decoupling in traditional methods, and supports the positioning of physical network element structures.
Smart Images

Figure CN115733724B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of communication network technology, and in particular to a method, device, electronic device and storage medium for locating the root cause of a service failure. Background Art
[0002] With the development of communication services, traditional business systems are gradually decoupled, and a complete business system includes multiple sub-business systems. In the case of complex topological relationships and call relationships in business systems, locating the root cause of business system failures has become a technical problem that needs to be solved urgently.
[0003] Currently, it is usually done by professional operation and maintenance personnel based on manual expert experience to locate the root cause of business failures, which requires a lot of manpower and high maintenance pressure, and needs to be based on the operation and maintenance personnel having sufficient experience in the business system. However, existing technologies cannot quickly and accurately locate the core root cause nodes and discover the core alarm root causes. Summary of the invention
[0004] The present invention provides a service fault root cause location method, device, electronic device and storage medium, which are used to solve the defect that the root cause location cannot be performed quickly and accurately in the prior art, and realize fast and accurate root cause location.
[0005] In a first aspect, the present invention provides a method for locating a root cause of a service failure, comprising:
[0006] Determine the time when the target indicator of the target business is abnormal as the abnormal time;
[0007] Based on the information of the call chain of the target service in the target time period with the abnormal time as the end time, determining each root cause node on the call chain;
[0008] Based on the alarms corresponding to the root cause nodes and the topological relationship between the devices, a root cause alarm among the alarms corresponding to the root cause nodes is determined.
[0009] In one embodiment, the target indicators include: average delay and success rate.
[0010] In one embodiment, the information of the call chain of the target service in the target time period with the abnormal time as the end time, determining each root cause node on the call chain specifically includes:
[0011] Based on the information of the call chain of the target business within the target time period, determining each abnormal node on the call chain;
[0012] Based on the information of each abnormal node, the root cause nodes among the abnormal nodes are determined.
[0013] In one embodiment, the determining, based on the alarms corresponding to the root cause nodes and the topological relationship between the devices, the root cause alarms among the alarms corresponding to the root cause nodes specifically includes:
[0014] Based on a preset association rule, determining the association relationship score between each target alarm pair and the alarms corresponding to the two root cause nodes included in each target alarm pair;
[0015] Based on the target alarm pairs and the correlation scores of the alarms corresponding to the two root cause nodes included in each target alarm pair, a root cause alarm among the alarms corresponding to the root cause nodes is determined.
[0016] In one embodiment, the determining the root cause nodes in the abnormal nodes based on the information of the abnormal nodes specifically includes:
[0017] Determine each possible root cause node in each abnormal node based on the depth and failure rate of each abnormal node in the call chain;
[0018] Based on the information of the child nodes of each of the possible root cause nodes, the root cause nodes in the abnormal nodes are determined.
[0019] In one embodiment, determining the root cause alarm in the alarms corresponding to the root cause nodes based on the target alarm pairs and the correlation relationship scores between the alarms corresponding to the two root cause nodes included in each target alarm pair specifically includes:
[0020] Generate an undirected graph based on the target alarm pairs and the correlation relationship scores between the alarms corresponding to the two root cause nodes included in each target alarm pair;
[0021] Based on the maximum spanning tree of the undirected graph, a root cause alarm among the alarms corresponding to the root cause nodes is determined.
[0022] In one embodiment, generating an undirected graph based on the target alarm pairs and the correlation relationship scores between the alarms corresponding to the two root cause nodes included in each target alarm pair specifically includes:
[0023] Based on the alarm time of the alarm corresponding to each of the root cause nodes, the alarm corresponding to each of the root cause nodes is divided into a plurality of target events;
[0024] For each target event, an undirected graph is generated.
[0025] In a second aspect, the present invention provides a device for locating a root cause of a service failure, comprising:
[0026] An indicator detection module is used to determine the moment when the target indicator of the target business is abnormal as an abnormal moment;
[0027] A node location module, configured to determine each root cause node on the call chain based on information of the call chain of the target service within a target time period with the abnormal time as the end time;
[0028] The alarm locating module is used to determine the root cause alarm among the alarms corresponding to the root cause nodes based on the alarms corresponding to the root cause nodes and the topological relationship between the devices.
[0029] In a third aspect, the present invention provides an electronic device, comprising a processor and a memory storing a computer program, wherein the processor implements the steps of any one of the above-mentioned methods for locating the root cause of a service failure when executing the computer program.
[0030] In a fourth aspect, the present invention provides a processor-readable storage medium, wherein the processor-readable storage medium stores a computer program, and the computer program is used to enable the processor to execute the steps of any one of the above-mentioned methods for locating the root cause of a business failure.
[0031] The method, device, electronic device and storage medium for locating the root cause of a service failure provided by the present invention can realize fast and accurate root cause location by determining the moment when the target indicator of the target service is abnormal as the abnormal moment, and based on the information of the call chain of the target service in the target time period with the abnormal moment as the end moment, the root cause nodes on the call chain are indeed called, and the root cause alarms in the alarms corresponding to the root cause nodes are determined based on the alarms corresponding to the root cause nodes and the topological relationship between the devices. Furthermore, the application architecture of the service root cause location scenario is closely linked and has a clear hierarchical layout. By decoupling and analyzing the service scenarios, three more detailed independent scenarios can be obtained, which is helpful to clarify the actual business of each scenario and reduce the complexity of the algorithm of each scenario. The root cause location is faster and more accurate, avoiding the shortcomings of the traditional method that the logic of root cause location is not layered and decoupled, and it is difficult to achieve rapid location. It can realize effective monitoring of the service system, integrate operation management with operation and maintenance guarantee, and can locate based on the physical network element structure. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0033] Figure 1It is a flow chart of the service fault root cause location method provided by the present invention;
[0034] Figure 2 It is a schematic diagram of a call chain in the service fault root cause location method provided by the present invention;
[0035] Figure 3 It is a schematic diagram of the process of generating the maximum spanning tree in the service fault root cause locating method provided by the present invention;
[0036] Figure 4 It is one of the schematic diagrams of the alarm time in the service fault root cause location method provided by the present invention;
[0037] Figure 5 This is the second schematic diagram of the alarm time in the service fault root cause location method provided by the present invention;
[0038] Figure 6 It is a structural schematic diagram of a service fault root cause locating device provided by the present invention;
[0039] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0040] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0041] Combine the following Figure 1-Figure 6 The invention describes a service fault root cause location method, device, electronic device and storage medium.
[0042] Figure 1 FIG. 1 is a flow chart of the method for locating the root cause of a service failure provided by the present invention. Figure 1 The following describes a method for locating the root cause of a service failure provided by an embodiment of the present application. Figure 1 As shown, the method includes: step 101, determining the time when the target indicator of the target business is abnormal as the abnormal time.
[0043] Specifically, the execution subject of the service fault root cause locating method provided by the embodiment of the present invention is the service fault root cause locating device provided by the present invention.
[0044] The target business may be a certain business carried by the business system.
[0045] For a target business, the target indicators of the business can be detected to determine whether the target indicators of the business are normal.
[0046] Exemplarily, the target indicator of the target business may be detected periodically to determine whether the target indicator of the target business in the current period is normal.
[0047] Exemplarily, the target indicator of the business within the second target time period may be detected to determine whether the target indicator of the business within the second target time period is normal.
[0048] The duration of the second target time period may be set according to actual conditions, and the embodiment of the present invention does not specifically limit the specific duration of the second target time period.
[0049] The target indicator is an indicator used to characterize whether the service is normal. For example, the target indicator may include average delay or success rate, etc. The embodiment of the present invention does not specifically limit the specific target indicator. The target indicator may be commonly referred to as a "golden indicator".
[0050] When the target indicator within the second target time period of the target business falls within the normal range, it can be determined that the target indicator of the target business is normal and there is no abnormality; when the target indicator within the second target time period of the target business does not fall within the normal range, it can be determined that the target indicator of the target business is abnormal.
[0051] After determining that the target indicator of the target business is abnormal, the time when the target indicator of the target business is determined to be abnormal can be used as the abnormal time of the target business.
[0052] Step 102: Based on the information of the call chain of the target business in the target time period with the abnormal time as the end time, each root cause node on the call chain is confirmed.
[0053] Specifically, call chain technology is a commonly used method for running services, which provides a series of service nodes as a call chain in a certain order to provide users with various services.
[0054] like Figure 2 As shown in the figure, each call chain is composed of multiple service nodes (referred to as "nodes"). A complete call chain basically presents a one-way transmission relationship of the link. Each call chain has a unique identifier traceId to distinguish it from other call chains. According to the call logic of the call chain, the call time and whether the call is successful can be known. The call time and whether the call is successful can reflect the node failure information.
[0055] The call chain information refers to the process of completing a business call in the business system, in which the call information (time, interface, level, result) between service nodes is logged, and then all the logged data is connected into a tree chain, that is, a call chain data is generated.
[0056] The target business can have one or more call chains.
[0057] In the embodiment of the present invention, the call chain information of the target service in the target time period (the target time period is the first target time period) is obtained. The duration of the first target time period is preset, and the end time of the first target time period is the abnormal time determined by step 101.
[0058] The duration of the first target time period may be set according to actual conditions, and the embodiment of the present invention does not specifically limit the specific duration of the first target time period.
[0059] Optionally, the duration of the first target time period is 5 minutes.
[0060] In a call chain, if the bottom node takes a long time or fails to call, the time taken by the upper node will also increase or fail accordingly. The root cause of the call chain is transmitted from bottom to top, so the lower the node, the more likely it is to become the root cause node. Therefore, the root cause nodes can be determined based on the call time of each node on each call chain.
[0061] Step 103: Based on the alarms corresponding to the root cause nodes and the topological relationship between the devices, determine the root cause alarms among the alarms corresponding to the root cause nodes.
[0062] Specifically, based on the root cause nodes at a certain abnormal moment located in step 102, the topological relationship of the platform architecture of the associated business system is used to further locate the alarms corresponding to the root cause nodes, analyze the relationship between the alarms corresponding to the root cause nodes, and finally locate the root cause alarms.
[0063] The alarm corresponding to the root cause node refers to the alarm associated with the root cause node.
[0064] The embodiment of the present invention determines the moment when the target indicator of the target business is abnormal as the abnormal moment, and based on the information of the call chain of the target business in the target time period with the abnormal moment as the end moment, each root cause node on the call chain is actually called, and based on the alarm corresponding to each root cause node and the topological relationship between each device, the root cause alarm in the alarm corresponding to each root cause node is determined, so that the root cause location can be quickly and accurately located. Furthermore, the application architecture of the business root cause location scenario is closely linked and has a clear hierarchical layout. By decoupling and analyzing the business scenarios, three more detailed independent scenarios can be obtained, which helps to clarify the actual business of each scenario and reduce the complexity of the algorithm of each scenario. The root cause location is faster and more accurate, avoiding the shortcomings of the traditional method that the logic of root cause location is not layered and decoupled, and it is difficult to achieve rapid location. It can achieve effective monitoring of the business system, integrate operation management with operation and maintenance guarantee, and can locate based on the physical network element structure.
[0065] Based on the content of any of the above embodiments, the target indicators include: average delay and success rate.
[0066] Specifically, the target indicators may include average delay and success rate, and whether the target service is abnormal is determined by combining the average delay and success rate.
[0067] The absolute threshold detection method can be used to detect whether the average delay and success rate are abnormal. The specific logic for determining whether they are abnormal is as follows:
[0068] a. If the average delay >= average delay threshold and the success rate >= success rate threshold, the fault type of abnormal time consumption is returned;
[0069] b. If the average delay < the average delay threshold and 0 < success rate < success rate threshold, the fault type of abnormal success rate is returned;
[0070] c. If the average delay < the average delay threshold and the success rate == 0, the fault type of the database exception is returned;
[0071] d. If the average delay >= average delay threshold and 0 < success rate < success rate threshold, then the fault type of abnormal time consumption and success rate is returned;
[0072] e. If the average delay >= average delay threshold and the success rate == 0, the fault type of both time consumption and database abnormality is returned.
[0073] The average delay threshold avgTimeThreshold can be set according to actual conditions, for example, the average delay threshold is 1 second. The embodiment of the present invention does not specifically limit the specific value of the average delay threshold.
[0074] The success rate threshold successRate can be set according to actual conditions, for example, the success rate threshold is 0.9. The embodiment of the present invention does not specifically limit the specific value of the success rate threshold.
[0075] The embodiment of the present invention is different from a complex anomaly detection algorithm. Instead, it adopts a simple absolute threshold and classification idea to analyze the target indicators of the business. The method is fast, effective, and easy to adjust. The detailed classification can reflect the real business problems in different situations.
[0076] Based on the content of any of the above embodiments, based on the information of the call chain of the target business in the target time period with the abnormal moment as the end time, each root cause node on the call chain is determined, specifically including: based on the information of the call chain of the target business in the target time period, each abnormal node on the call chain is determined.
[0077] Specifically, for each call chain, it is determined whether the call chain is a call chain with abnormal call consumption based on the time consumption of the call chain.
[0078] The statistics of the call chain time consumption are normally distributed. The preset quantiles can be used to approximate the confidence interval to filter out the call chains with abnormal call consumption.
[0079] Preferably, the preset quantile may be the 95th quantile, which is used to approximate the data interval with a confidence interval of 5%. If the time consumption of the call chain belongs to the above 5% data interval, the call chain is an abnormal call chain.
[0080] Optionally, the preset quantile may also be the 90th quantile, and the 90th quantile is used to approximate the data interval with a confidence interval of 10%. If the time consumption of the call chain belongs to the above 10% data interval, the call chain is an abnormal call chain.
[0081] The nodes on the call chain with abnormal call time consumption can be traversed by searching to determine the abnormal nodes.
[0082] The call time of the same type of nodes on the call chain is based on normal statistical distribution.
[0083] If the node time consumption meets the abnormal threshold of the node type or the node call fails, the node is an abnormal node; if the node time consumption does not meet the abnormal threshold of the node type or the node call succeeds, but the node's parent node time consumption is more than twice the total time consumption of all nodes at the same level of the node, then the node is a suspected abnormal node.
[0084] by Figure 2 Taking the partial call chain in as an example, the search process of abnormal nodes in the call chain is as follows:
[0085] Starting from the head node, record osb_001 as the abnormal node, then record the node information {'osb_001':[1,1,0],}(node name:[number of abnormalities, depth, suspected abnormality (abnormal 0, suspected abnormality 1, normal 2)]);
[0086] Continue to search down to the csf_001 node;
[0087] If the time taken by the csf_001 node meets the abnormal threshold of csf_001, the node information {'osb_001':[1,1,0],'csf_001':[1,2,0],} is recorded;
[0088] Knowing that csf_001 is an abnormal node, continue searching down to csf_003 node;
[0089] If the csf_003 node time consumption does not meet the csf_003 abnormal threshold but is a node that failed to call, then the node information {'osb_001':[1,1,0], 'csf_001':[1,2,0],
[0090] 'csf_003': [1,3,0],};
[0091] Knowing that csf_003 is an abnormal node, continue to search down to local_15 and local_16 nodes;
[0092] If the local_15 node time consumption does not meet the local_15 abnormal threshold but is a node that fails to call, then the node information {'osb_001': [1,1,0], 'csf_001': [1,2,0], 'csf_003': [1,3,0], 'local_15': [1,4,0]} is recorded;
[0093] The time consumption of local_16 node does not meet the abnormal threshold of local_16 and the call is successful, so local_16 is not an abnormal node; the time consumption of csf_003 is 20s, and the time consumption of csf_003 subnodes is self-call, local_15, and local_16 (4s, 3s, and 2s), which is more than twice the time consumption of subnode call. Therefore, local_16 is a suspected abnormal node, and the node information is recorded {'osb_001': [1,1,0], 'csf_001': [1,2,0], 'csf_003': [1,3,0], 'local_15': [1,4,0] 'local_16': [1,4,1]};
[0094] Knowing that local_15 is an abnormal node, continue to search down to the db_003 node of the jdbc layer;
[0095] The time consumption of db_003 node does not meet the abnormal threshold of db_003 and the call is successful, so local_16 is not an abnormal node; the time consumption of local_15 is 3s and the time consumption of local_15 child node is db_003 1s, which is more than twice the time consumption of child node call, so db_003 is a suspected abnormal node, and the node information is recorded {'osb_001': [1,1,0], 'csf_001': [1,2,0], 'csf_003': [1,3,0], 'local_15': [1,4,0] 'local_16': [1,4,1], 'db_003': [1,5,1]};
[0096] It is known that local_16 is a suspected abnormal node, so the search is stopped;
[0097] When a global search is required, it is known that local_16 is a suspected abnormal node and has child nodes;
[0098] Continue searching, and all subsequent child nodes are judged by calling status and the influence of delay is ignored. For example, the db_003 node will not be recorded as an exception when called by local_16, so the recorded db_003 node information {'osb_001':[1,1,0], 'csf_001':[1,2,0], 'csf_003':[1,3,0], 'local_15':[1,4,0] 'local_16':[1,4,1], 'db_003':[1,5,1]} is still consistent with the recorded information under the local_15 call, and repeat this process to complete the entire search.
[0099] Through the above process, each abnormal node can be determined.
[0100] Based on the information of each abnormal node, each root cause node in each abnormal node is determined.
[0101] Optionally, if the bottom-level node consumes more time or fails to call, the time consumption of the upper-level node will also increase or fail accordingly. The root cause node in the abnormal node can be determined by analyzing whether the abnormal node consumes more time or fails to call due to its own reasons or due to the reasons of the descendant nodes of the abnormal node.
[0102] If the abnormal node takes a long time or the call failure is caused by its own reasons, the abnormal node can be determined as the root cause node; if the abnormal node takes a long time or the call failure is caused by the descendant nodes of the abnormal node, the abnormal node is not determined as the root cause node.
[0103] The embodiment of the present invention is different from the method of generating an abnormal ID for abnormal event data and storing the mapping relationship between abnormal events and request IDs. Instead, it effectively utilizes the conditions of node information inheritance, analyzes the abnormal state of the node child by child, and then determines the root cause by the number of failures and the failure rate, and adopts a pruning strategy to speed up the search speed, so that the root cause node can be located more quickly and accurately. Furthermore, it avoids the shortcomings of the traditional method that the fault propagation path needs to be combed afterwards and it is difficult to quickly optimize the topology architecture.
[0104] Based on the content of any of the above embodiments and based on the information of each abnormal node, determining each root cause node in each abnormal node specifically includes: determining each possible root cause node in each abnormal node based on the depth and failure rate of each abnormal node in the call chain.
[0105] Optionally, the number of call chains to be counted is n 调用链 , the total number of calls of a node in a certain layer (the depth of the layer is deep) is n 节点 , and the number of failures is m 故障 , the number of suspected failures is l 疑似 . You can start the analysis from the deepest abnormal node.
[0106] If m 故障 >=0.8*(deep-2)*0.05*n 调用链 And the node failure rate p 故障率 If the fault rate is greater than the fault rate threshold, the node is a possible root cause node.
[0107] Or, if m 故障 >=0.8*(deep-2)*0.1*n 调用链 And the node failure rate p 故障率 If the fault rate is greater than the fault rate threshold, the node is a possible root cause node.
[0108] p 故障率 =m 故障 / n 节点
[0109] The failure rate threshold can be set according to actual conditions. The embodiment of the present invention does not specifically limit the specific value of the failure rate threshold.
[0110] Preferably, the failure rate threshold is set to 0.5 by default.
[0111] Based on the information of the child nodes of each possible root cause node, each root cause node in each abnormal node is determined.
[0112] Optionally, if a layer has a single possible root cause node, it is directly used as the root cause node. If a layer has multiple possible root cause nodes, the possible root cause node with a larger number of failures is returned as the root cause node; if a layer has multiple possible root cause nodes and the number of failures is the same, the possible root cause node with a higher failure rate is returned as the root cause node; if the number of failures is equal to the failure rate, the docker node is returned as the root cause node.
[0113] It should be noted that if there is a service application call chain topology map, in which service X calls four application nodes A, B, C, and D, but in the actual search process, service X only calls three application nodes A, B, and C, then node D is returned as the lost node.
[0114] According to the node information obtained from the call chain search statistics, the topological relationship between the business node and the server node is integrated, and finally the following abnormal statistical node information of the server node is obtained ({depth: {2:{'call node': total number of call nodes}, 1:{'suspected abnormal node': number of suspected abnormal nodes}, 0:{'abnormal node': number of abnormal nodes}}}).
[0115] For example, the actual number of search call chains is 102. The analysis starts from the deepest fifth layer. The fifth layer has default (which means that the node does not exist) and the number of docker_006 failures satisfies 108*0.1*0.8*3 and the failure rate is greater than the failure rate threshold. Based on the principle that the deepest node is most likely to be the root cause, the analysis is stopped and the docker_006 node in the fifth layer is directly located. (It can be seen that the data of the other layers meet the failure number but not the failure rate, which can be used as a verification to some extent, indicating that the real root cause node is the docker_006 node).
[0116] The embodiment of the present invention determines each possible root cause node in each abnormal node and determines each root cause node based on the information of the child nodes of each possible root cause node, so that the root cause node can be located more quickly and accurately.
[0117] Based on the content of any of the above embodiments, based on the alarms corresponding to each root cause node, and the topological relationship between each device, determine the root cause alarms among the alarms corresponding to each root cause node, specifically including: based on preset association rules, determine the association relationship scores between each target alarm pair and the alarms corresponding to the two root cause nodes included in each target alarm pair.
[0118] Optionally, based on a preset association rule, frequently occurring alarm pairs among the alarms corresponding to each root cause node are determined, and each alarm pair includes two alarms.
[0119] Device refers to the device where the root node is located.
[0120] Based on the preset association rules, the association score between the two alarms included in each alarm pair may also be determined.
[0121] The association rules may be obtained based on historical alarm data and real-time alarm data streams.
[0122] It should be noted that the primary and secondary rule table (i.e., the preset association rules) can be continuously updated based on historical alarm data and real-time alarm data streams to explore new rules between alarms and eliminate erroneous association rules, thereby improving the accuracy of target alarm determination and association relationship score acquisition, thereby locating the root cause alarm more quickly and accurately.
[0123] Based on each target alarm pair and the correlation relationship scores of the alarms corresponding to the two root cause nodes included in each target alarm pair, the root cause alarm among the alarms corresponding to the root cause nodes is determined.
[0124] Optionally, based on the correlation score between the two alarms included in each alarm pair, the primary and secondary relationship between the alarms corresponding to each root cause node can be determined, so that the relatively primary alarm among them can be determined as the root cause alarm.
[0125] The embodiment of the present invention determines the correlation score between each target alarm pair and the alarms corresponding to the two root cause nodes included in each target alarm pair based on a preset correlation rule, and determines the root cause alarm among the alarms corresponding to each root cause node based on the correlation score between each target alarm pair and the alarms corresponding to the two root cause nodes included in each target alarm pair, thereby locating the root cause alarm more quickly and accurately.
[0126] Based on the content of any of the above embodiments, based on the correlation scores between each target alarm pair and the alarms corresponding to the two root cause nodes included in each target alarm pair, the root cause alarms in the alarms corresponding to each root cause node are determined, specifically including: based on each target alarm pair and the correlation scores between the alarms corresponding to the two root cause nodes included in each target alarm pair, an undirected graph is generated.
[0127] Specifically, the vertices in the undirected graph represent the alarms corresponding to the root cause nodes, the edges in the undirected graph represent that the alarms represented by the two vertices connected by the edge constitute a target alarm pair, and the weight of the edge is the correlation score between the alarms, thereby generating an undirected graph. The undirected graph in the embodiment of the present invention may be referred to as an alarm graph.
[0128] Based on the maximum spanning tree of the undirected graph, a root cause alarm among the alarms corresponding to each root cause node is determined.
[0129] Specifically, the maximum spanning tree algorithm is similar to the minimum spanning tree algorithm. The purpose is to select a subset E' of the edge set E in a given undirected graph G(V,E), where each (u,v) is assigned a weight w, so that all points in the point set V are connected by edges in E', and the sum of all edge weights in the set E' is maximized.
[0130] Specifically, Kruskal's Algorithm or Prim's Algorithm can be used. Both algorithms apply Greedy's idea. Taking Kruskal's Algorithm as an example, each time an edge (u, v) with the largest weight is selected to determine whether it forms a cycle with the edge in E'; if it does not form a cycle, (u, v) is added to E'; otherwise, the edge (u, v) is deleted; this step is repeated until all points in the graph are connected by the tree. The specific implementation process of the maximum spanning tree of the undirected graph composed of vertices 1-6 is as follows Figure 3 shown.
[0131] After the maximum spanning tree of the undirected graph is obtained, the alarm represented by the vertex with an in-degree of 0 can be determined as the root cause alarm.
[0132] The embodiment of the present invention generates an undirected graph based on the correlation scores between each target alarm pair and the alarms corresponding to the two root cause nodes included in each target alarm pair, and determines the root cause alarms among the alarms corresponding to each root cause node based on the maximum spanning tree of the undirected graph, so as to locate the root cause alarms more quickly and accurately.
[0133] Based on the content of any of the above embodiments, an undirected graph is generated based on the correlation scores between each target alarm pair and the alarms corresponding to the two root cause nodes included in each target alarm pair, specifically including: based on the alarm time of the alarm corresponding to each root cause node, the alarm corresponding to each root cause node is divided into several target events.
[0134] Specifically, based on the alarm time of the alarm corresponding to each root cause node, the high frequency time of the same alarm and the correlation time between different alarms can be obtained.
[0135] The high frequency time of the same alarm refers to the average interval between the second or multiple alarms of a certain alarm. The high frequency time is used to aggregate the same alarms with similar occurrence time during the event aggregation process.
[0136] By considering different devices and different time periods, high-frequency time is weighted.
[0137] The high frequency time is calculated as follows:
[0138] First, a preliminary calculation of the high-frequency time is performed on different devices based on the alarm start time and end time, that is, the overlapping time intervals are merged; then the first occurrence time of the alarm is subtracted from the last occurrence time; finally, weighted processing is performed based on the different times the alarm is divided into the overlapping interval and the information on different devices.
[0139] by Figure 4 Taking the time period of alarm A as an example, the high-frequency time on device A1 is: st3-st1, st5-st4, and there are 2 high-frequency time periods; the high-frequency time on device A2 is: st4-st3, and there is 1 high-frequency time period.
[0140] The calculation formula for the high frequency time of alarm A is:
[0141]
[0142] Here, 5 / 5 and 2 / 4 represent the weights of device A1 and device A2 respectively.
[0143] The correlation time between different alarms refers to the length of time that a certain alarm is associated with other alarms. The correlation time is used to aggregate two different alarms with similar occurrence times during the event aggregation process.
[0144] By considering different devices and different time periods, weighted processing of the association time is performed.
[0145] The correlation time is calculated as follows:
[0146] First, a preliminary calculation of the correlation time is performed on different devices based on the start time and the end time, that is, the overlapping secondary alarm time intervals are merged; then the occurrence time of the secondary alarm is subtracted from the occurrence time of the most recent main alarm; finally, weighted processing is performed based on the different times that the alarm is divided into the main alarm interval and the information on different devices.
[0147] by Figure 5 Taking the alarm shown as an example, the two alarms associated with alarm B on device A1 are B1 and B2, the associated time is st1B-st4, st3B-st5, and the number of associations is 2. The weighted average of the associated alarm events on all devices will tend to favor the device with frequent A alarms.
[0148] According to the topological relationship between devices, the high frequency time of the same alarm and the correlation time between different alarms, multiple alarms with correlation can be classified into the same target event and several target events can be determined.
[0149] It should be noted that the calculation method of high-frequency time and correlation time can also be continuously learned through historical data, so that the accuracy of event division can be improved in the process of continuous iteration.
[0150] For each target event, an undirected graph is generated.
[0151] Specifically, for each target event, an undirected graph corresponding to the target event is generated based on the target alarm pairs included in the target event and the correlation relationship scores between the alarms corresponding to the two root cause nodes included in each target alarm pair.
[0152] The process of generating an undirected graph can be referred to in the above-mentioned embodiment, which will not be described again here.
[0153] The embodiment of the present invention divides the alarms corresponding to each root cause node into a plurality of target events based on the alarm time of the alarms corresponding to each root cause node, and can locate the root cause alarm more quickly and accurately for each target event.
[0154] The service fault root cause locating device provided by the present invention is described below. The service fault root cause locating device described below and the service fault root cause locating method described above can be referenced to each other.
[0155] Figure 6 Schematic diagram of the structure of the service fault root cause location device provided by the present invention. Based on the content of any of the above embodiments, Figure 6 As shown, the service fault root cause location device includes an indicator detection module 601, a node location module 602 and an alarm location module 603, wherein:
[0156] The indicator detection module 601 is used to determine the moment when the target indicator of the target business is abnormal as the abnormal moment;
[0157] A node location module 602 is used to determine each root cause node on the call chain based on information of the call chain of the target service in the target time period with the abnormal time as the end time;
[0158] The alarm locating module 603 is used to determine the root cause alarm among the alarms corresponding to the root cause nodes based on the alarms corresponding to the root cause nodes and the topological relationship between the devices.
[0159] Specifically, the indicator detection module 601, the node positioning module 602 and the alarm positioning module 603 are electrically connected in sequence.
[0160] For the target business, the indicator detection module 601 can detect the target indicator of the business within the second target time period, and determine whether the target indicator of the business within the second target time period is normal.
[0161] When the target indicator within the second target time period of the target business falls within the normal range, the indicator detection module 601 can determine that the target indicator of the target business is normal and there is no abnormality; when the target indicator within the second target time period of the target business does not fall within the normal range, the indicator detection module 601 can determine that the target indicator of the target business is abnormal.
[0162] The node location module 602 obtains the information of the call chain of the target business within the target time period, analyzes the call time of each node on each call chain, and determines each root cause node.
[0163] The alarm locating module 603 further locates the alarms corresponding to the root cause nodes based on the located root cause nodes at a certain abnormal moment and the topological relationship of the platform architecture of the associated business system, analyzes the relationship between the alarms corresponding to the root cause nodes, and finally locates the root cause alarms.
[0164] Optionally, the target indicators include: average delay and success rate.
[0165] Optionally, the node positioning module 602 may include:
[0166] An abnormal node determination unit, used to determine each abnormal node on the call chain based on the call chain information of the target business within the target time period;
[0167] The root cause node determination unit is used to determine each root cause node in each abnormal node based on the information of each abnormal node.
[0168] Optionally, the alarm locating module 603 may include:
[0169] An association unit, configured to determine, based on a preset association rule, an association relationship score between each target alarm pair and the alarms corresponding to the two root cause nodes included in each target alarm pair;
[0170] The positioning unit is used to determine the root cause alarm among the alarms corresponding to the root cause nodes based on the correlation relationship scores of the target alarm pairs and the alarms corresponding to the two root cause nodes included in each target alarm pair.
[0171] Optionally, the root cause node determination unit may include:
[0172] A possible root cause determination subunit, used to determine each possible root cause node in each abnormal node based on the depth and failure rate of each abnormal node in the call chain;
[0173] The root cause node determination subunit is used to determine each root cause node in each abnormal node based on the information of the subnodes of each possible root cause node.
[0174] Optionally, the positioning unit may include:
[0175] A graph generation subunit, for generating an undirected graph based on the correlation relationship scores between each target alarm pair and the alarms corresponding to the two root cause nodes included in each target alarm pair;
[0176] The alarm locating subunit is used to determine the root cause alarm among the alarms corresponding to the root cause nodes based on the maximum spanning tree of the undirected graph.
[0177] Optionally, the graph generating subunit is specifically used to divide the alarms corresponding to each root cause node into a plurality of target events based on the alarm time of the alarms corresponding to each root cause node; and generate an undirected graph for each target event respectively.
[0178] The service fault root cause locating device provided in the embodiment of the present invention is used to execute the service fault root cause locating method of the present invention. Its implementation method is consistent with the implementation method of the service fault root cause locating method provided by the present invention and can achieve the same beneficial effects, which will not be repeated here.
[0179] The service fault root cause location device is used in the service fault root cause location method in the above embodiments. Therefore, the description and definition in the service fault root cause location method in the above embodiments can be used for understanding each execution module in the embodiments of the present invention.
[0180] The embodiment of the present invention determines the moment when the target indicator of the target business is abnormal as the abnormal moment, and based on the information of the call chain of the target business in the target time period with the abnormal moment as the end moment, each root cause node on the call chain is actually called, and based on the alarm corresponding to each root cause node and the topological relationship between each device, the root cause alarm in the alarm corresponding to each root cause node is determined, so that the root cause location can be quickly and accurately located. Furthermore, the application architecture of the business root cause location scenario is closely linked and has a clear hierarchical layout. By decoupling and analyzing the business scenarios, three more detailed independent scenarios can be obtained, which helps to clarify the actual business of each scenario and reduce the complexity of the algorithm of each scenario. The root cause location is faster and more accurate, avoiding the shortcomings of the traditional method that the logic of root cause location is not layered and decoupled, and it is difficult to achieve rapid location. It can achieve effective monitoring of the business system, integrate operation management with operation and maintenance guarantee, and can locate based on the physical network element structure.
[0181] The electronic device and storage medium provided by the present invention are described below. The electronic device and storage medium described below and the service fault root cause locating method described above can be referenced to each other.
[0182] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the computer program in the memory 730 to execute the steps of the service fault root cause location method, for example, including: determining the moment when the target indicator of the target service is abnormal as the abnormal moment; based on the information of the call chain of the target service in the target time period with the abnormal moment as the end moment, the root cause nodes on the call chain are indeed called; based on the alarms corresponding to the root cause nodes and the topological relationship between the devices, the root cause alarms in the alarms corresponding to the root cause nodes are determined.
[0183] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0184] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the steps of the service fault root cause locating method provided by the above methods, for example, including: determining the moment when an abnormality exists in the target indicator of the target service as the abnormal moment; based on the information of the call chain of the target service in the target time period with the abnormal moment as the end moment, actually calling each root cause node on the call chain; based on the alarm corresponding to each root cause node and the topological relationship between each device, determining the root cause alarm in the alarm corresponding to each root cause node.
[0185] On the other hand, an embodiment of the present application also provides a processor-readable storage medium, which stores a computer program, and the computer program is used to enable the processor to execute the steps of the methods provided by the above embodiments, for example, including: determining the moment when an abnormality exists in the target indicator of the target business as the abnormal moment; based on the information of the call chain of the target business in the target time period with the abnormal moment as the end moment, actually calling each root cause node on the call chain; based on the alarm corresponding to each root cause node and the topological relationship between each device, determining the root cause alarm in the alarm corresponding to each root cause node.
[0186] The processor-readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NANDFLASH), solid-state drive (SSD)), etc.
[0187] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.
[0188] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0189] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for locating the root cause of a service failure. It is characterized in that include: Determine the time when the target indicator of the target business is abnormal as the abnormal time; Based on the information of the call chain of the target service in the target time period with the abnormal time as the end time, determining each root cause node on the call chain; Determine the root cause alarm among the alarms corresponding to the root cause nodes based on the alarms corresponding to the root cause nodes and the topological relationship between the devices; The determining, based on the alarms corresponding to the root cause nodes and the topological relationship between the devices, the root cause alarms among the alarms corresponding to the root cause nodes specifically includes: Based on a preset association rule, determining the association relationship score between each target alarm pair and the alarms corresponding to the two root cause nodes included in each target alarm pair; Determine the root cause alarm among the alarms corresponding to the root cause nodes based on the target alarm pairs and the correlation relationship scores of the alarms corresponding to the two root cause nodes included in each target alarm pair; The determining the root cause alarm among the alarms corresponding to the root cause nodes based on the target alarm pairs and the correlation relationship scores between the alarms corresponding to the two root cause nodes included in each target alarm pair specifically includes: Generate an undirected graph based on the target alarm pairs and the correlation relationship scores between the alarms corresponding to the two root cause nodes included in each target alarm pair; Determine, based on the maximum spanning tree of the undirected graph, a root cause alarm among the alarms corresponding to the root cause nodes; The generating of an undirected graph based on the target alarm pairs and the correlation relationship scores between the alarms corresponding to the two root cause nodes included in each target alarm pair specifically includes: Based on the alarm time of the alarm corresponding to each of the root cause nodes, the alarm corresponding to each of the root cause nodes is divided into a plurality of target events; For each target event, generate an undirected graph; Based on the alarm time of the alarm corresponding to each of the root cause nodes, the alarm corresponding to each of the root cause nodes is divided into a plurality of target events, including: Based on the alarm time of the alarm corresponding to each root cause node, the high frequency time of the same alarm and the correlation time between different alarms are obtained. The high frequency time of the same alarm refers to the average interval between the occurrence of the same alarm twice or more times. The high frequency time of the same alarm is used to aggregate the same alarms with similar occurrence time during the event aggregation process. The correlation time between different alarms refers to the correlation duration between an alarm and other alarms. The correlation time between different alarms is used to aggregate different alarms with similar occurrence time during the event aggregation process. According to the topological relationship between the devices, the high frequency time of the same alarm and the correlation time between the different alarms, a plurality of alarms having correlation are classified into the same target event, and a plurality of target events are determined.
2. The method for locating the root cause of a service failure according to claim 1, It is characterized in that The target indicators include: average delay and success rate.
3. The method for locating the root cause of a service failure according to claim 1, It is characterized in that The information of the call chain of the target service in the target time period with the abnormal time as the end time is used to determine each root cause node on the call chain, specifically including: Based on the information of the call chain of the target business within the target time period, determining each abnormal node on the call chain; Based on the information of each abnormal node, the root cause nodes among the abnormal nodes are determined.
4. The method for locating the root cause of a service failure according to claim 3, It is characterized in that The determining, based on the information of the abnormal nodes, the root cause nodes in the abnormal nodes specifically includes: Determine each possible root cause node in each abnormal node based on the depth and failure rate of each abnormal node in the call chain; Based on the information of the child nodes of each of the possible root cause nodes, the root cause nodes in the abnormal nodes are determined.
5. A device for locating the root cause of a business failure, It is characterized in that include: An indicator detection module is used to determine the moment when the target indicator of the target business is abnormal as an abnormal moment; A node location module, configured to determine each root cause node on the call chain based on information of the call chain of the target service within a target time period with the abnormal time as the end time; An alarm locating module, configured to determine a root cause alarm among the alarms corresponding to the root cause nodes based on the alarms corresponding to the root cause nodes and the topological relationship between the devices; The determining, based on the alarms corresponding to the root cause nodes and the topological relationship between the devices, the root cause alarms among the alarms corresponding to the root cause nodes specifically includes: Based on a preset association rule, determining the association relationship score between each target alarm pair and the alarms corresponding to the two root cause nodes included in each target alarm pair; Determine the root cause alarm among the alarms corresponding to the root cause nodes based on the target alarm pairs and the correlation relationship scores of the alarms corresponding to the two root cause nodes included in each target alarm pair; The determining the root cause alarm among the alarms corresponding to the root cause nodes based on the target alarm pairs and the correlation relationship scores between the alarms corresponding to the two root cause nodes included in each target alarm pair specifically includes: Generate an undirected graph based on the target alarm pairs and the correlation relationship scores between the alarms corresponding to the two root cause nodes included in each target alarm pair; Determine, based on the maximum spanning tree of the undirected graph, a root cause alarm among the alarms corresponding to the root cause nodes; The generating of an undirected graph based on the target alarm pairs and the correlation relationship scores between the alarms corresponding to the two root cause nodes included in each target alarm pair specifically includes: Based on the alarm time of the alarm corresponding to each of the root cause nodes, the alarm corresponding to each of the root cause nodes is divided into a plurality of target events; For each target event, generate an undirected graph; Based on the alarm time of the alarm corresponding to each of the root cause nodes, the alarm corresponding to each of the root cause nodes is divided into a plurality of target events, including: Based on the alarm time of the alarm corresponding to each root cause node, the high frequency time of the same alarm and the correlation time between different alarms are obtained. The high frequency time of the same alarm refers to the average interval between the occurrence of the same alarm twice or more times. The high frequency time of the same alarm is used to aggregate the same alarms with similar occurrence time during the event aggregation process. The correlation time between different alarms refers to the correlation duration between an alarm and other alarms. The correlation time between different alarms is used to aggregate different alarms with similar occurrence time during the event aggregation process. According to the topological relationship between the devices, the high frequency time of the same alarm and the correlation time between the different alarms, a plurality of alarms having correlation are classified into the same target event, and a plurality of target events are determined.
6. An electronic device comprising a processor and a memory storing a computer program, It is characterized in that When the processor executes the computer program, the steps of the service fault root cause locating method according to any one of claims 1 to 4 are implemented.
7. A processor-readable storage medium, It is characterized in that The processor-readable storage medium stores a computer program, and the computer program is used to enable the processor to execute the steps of the service fault root cause locating method according to any one of claims 1 to 4.
