A method and device for root cause analysis

Abstract

The embodiment of the present invention discloses a method and device for root cause analysis, the method comprising: after monitoring that a first service node processes a first service and generates an abnormality, obtaining a first indicator curve in which an abnormality occurs in the first service node, determining a second service node corresponding to the first service, determining a preliminary abnormal node from each second service node, wherein the preliminary abnormal node has a second indicator curve matching the first indicator curve, and determining a root cause node causing the abnormality from the first service node and the preliminary abnormal node. The present invention uses the node having the second indicator curve matching the first indicator curve as the preliminary abnormal node, that is, judging the similarity according to the curve trend, so as to efficiently perform root cause analysis while improving the accuracy of the root cause analysis result.

Description

Technical Field

[0001] The present invention relates to the field of financial technology (Fintech) computers, and in particular to a method and device for root cause analysis. Background Art

[0002] Root Cause Analysis (RCA) refers to finding the root cause of the problem from a large amount of relevant abnormal information when a system exception occurs. RCA analysis is to find the root cause of the problem through the relationship between some data and data reflected by the system. For example: a host disk exception, a command execution time of a system is too long, a method has a null pointer error, etc.

[0003] There is no universal solution for RCA. When the system reaches a certain scale, there will be many subsystems, relationships, indicators, and alarms. At this time, it will be very difficult to perform RCA analysis.

[0004] At present, the commonly used RCA solution is alarm analysis: when business anomalies occur, a complete monitoring system generally has a large number of monitoring alarms. Some of these alarms are based on indicator thresholds, some are based on error keyword logs, and some are based on system module calls. These alarms generally use the data collected by themselves as monitoring points. When business anomalies occur, they may only be affected by other system anomalies and may not be the root cause of the problem. In other words, when business anomalies occur, there may be a large number of alarms, some of which may be the root cause, and others may be related to the anomaly, but not the root cause of the problem. Alarm-based RCA analysis is to remove high-frequency alarms and useless alarms from the numerous alarms, sort out the dependencies, and finally determine the root cause of the problem.

[0005] The alarm analysis proposed in the above solution is more about looking at the occurrence of problems from a micro perspective. For example, a transaction is interrupted, an interface reports an error, a failure log is output, etc. This kind of problem analysis from a micro perspective requires a lot of analysis, which makes it very inefficient to find the root cause when an exception occurs. Summary of the invention

[0006] The embodiments of the present invention provide a method and device for root cause analysis, so as to solve the problem of how to perform root cause analysis efficiently and accurately.

[0007] In a first aspect, an embodiment of the present invention provides a method for root cause analysis, comprising:

[0008] After monitoring that the first service node generates an abnormality in processing the first service, obtaining a first indicator curve in which the abnormality occurs in the first service node;

[0009] Determine a second service node corresponding to the first service, where the second service node is any service node among the service nodes processing the first service except the first service node;

[0010] Determine a preliminarily selected abnormal node from each second service node, wherein the preliminarily selected abnormal node has a second indicator curve matching the first indicator curve;

[0011] A root cause node causing the exception is determined from the first service node and the preliminarily selected exception nodes.

[0012] The above scheme, after monitoring an abnormality in a business node, matches the first indicator curve where the abnormality occurs with the indicator curves of other business nodes processing the first business, and uses the business node with the second indicator curve matching the first indicator curve as the preliminary abnormal node, that is, judging the similarity based on the curve trend, and then determining the root cause node from the preliminary abnormal node and the first business node, thereby improving the accuracy of the root cause analysis results while efficiently performing root cause analysis.

[0013] Optionally, determining a preliminarily selected abnormal node from each second service node includes:

[0014] For each second service node, determine the indicator curve associated with the first service among the indicator curves of the second service node; if there is a second indicator curve matching the first indicator curve among the indicator curves associated with the first service, determine the second service node as a preliminarily selected abnormal node.

[0015] In the above scheme, the indicator curve associated with the first business is determined from the indicator curves of the second business node corresponding to the first business, and the indicator curve associated with the first business is matched; in actual business, a business node may involve multiple businesses, and a business node may be provided with many indicator curves based on various requirements; in this scheme, the efficiency of the root cause analysis can be improved by narrowing the scope of the business nodes and the scope of the indicator curves; at the same time, since irrelevant curves are excluded, the matching degree of some curves with the first indicator curve due to fluctuations is avoided, thereby improving the accuracy of the root cause analysis.

[0016] Optionally, before monitoring that the first service node processes the first service abnormally, the method further includes:

[0017] Determine, through a service processing flow of the third service, each third service node for processing the third service, and determine, among each indicator curve of each third service node, an indicator curve associated with the third service; the third service is any one of the services;

[0018] A comparison relationship among a service identifier of the third service, each third service node, and an indicator curve associated with the third service in each third service node is generated.

[0019] The above solution avoids the problem of too large a scope of similarity judgment. In this solution, the business nodes that each business passes through during processing and the indicator curves involved in the business nodes are sorted out, so that when making similarity judgments, only the business nodes and indicator curves involved in the business are found in the comparison relationship. In this way, the set of similarity judgments is greatly reduced, and the accuracy is improved.

[0020] Optionally, determining whether the second indicator curve matches the first indicator curve in the following manner includes:

[0021] Determine a first ranking of the first abnormal start time, the first abnormal peak time, and the indicator fluctuation values ​​at each time in the abnormal period in the first indicator curve;

[0022] Determine a second ranking of the second abnormal start time, the second abnormal peak time, and the indicator fluctuation values ​​at each time in the abnormal period in the second indicator curve;

[0023] According to a first matching degree between the first abnormality start time and the second abnormality start time, a second matching degree between the first abnormality peak time and the second abnormality peak time, and a third matching degree between the first sorting and the second sorting, it is determined whether the matching degree between the first indicator curve and the second indicator curve is greater than a set threshold.

[0024] The above scheme can obtain the similarity between a certain curve and the abnormal indicator curve by observing the matching degree from three dimensions: the abnormal start time, the abnormal peak time and the index fluctuation value. When it is higher than a certain threshold, it is determined as the primary abnormal node. By calculating the matching degree, the accuracy of the root cause analysis results is improved.

[0025] Optionally, determining the root cause node causing the exception from the first service node and the preliminarily selected exception node includes:

[0026] For the same indicator where an abnormality occurs, determining a first change amount of the indicator by the first service node and a second change amount of the indicator by the preliminarily selected abnormal node;

[0027] If the relationship between the first change amount and the second change amount does not meet the set condition, excluding the initially selected abnormal node;

[0028] A root cause node causing the exception is determined from the preliminarily selected exception nodes and the first service node after the exclusion operation.

[0029] The above method can exclude some curves that were originally judged to be similar but unreasonable through the relationship between the first variation and the second variation, thereby improving the accuracy of the judgment.

[0030] Optionally, determining a root cause node causing the exception from the initially selected abnormal nodes and the first service node after the exclusion operation includes:

[0031] For the indicator corresponding to the first indicator curve, determine the influencing party on the indicator when processing the first business from the preliminarily selected abnormal nodes and the first business node, and determine the influencing party as the root cause node causing the abnormality.

[0032] The above scheme determines the final root cause node according to the dependency relationship between the initially selected abnormal node and the abnormal node, thereby reducing the possibility of misjudgment and improving the accuracy of root cause analysis.

[0033] Optionally, the method further includes:

[0034] Determining a first attribute associated with the indicator corresponding to the first indicator curve and a second attribute associated with the indicator corresponding to the second indicator curve;

[0035] The same attributes of the first attributes and the second attributes are displayed.

[0036] The above solution can confine the anomalies to a smaller range or category by finding all indicators with similar changes and the attributes associated with the indicators and performing statistics and outputting them, thereby improving the readability of the root cause analysis results.

[0037] In a second aspect, an embodiment of the present invention provides a root cause analysis device, including:

[0038] An acquisition module, configured to acquire a first indicator curve in which the first service node has an abnormality after monitoring that the first service node has an abnormality in processing the first service;

[0039] A processing module, used to determine a second service node corresponding to the first service, where the second service node is any service node among the service nodes processing the first service except the first service node;

[0040] The processing module is further used to determine a preliminarily selected abnormal node from each second service node, wherein the preliminarily selected abnormal node has a second indicator curve matching the first indicator curve;

[0041] The processing module is further used to determine the root cause node causing the exception from the first service node and the preliminarily selected exception nodes.

[0042] Optionally, the processing module is specifically used for:

[0043] For each second service node, determine the indicator curve associated with the first service among the indicator curves of the second service node; if there is a second indicator curve matching the first indicator curve among the indicator curves associated with the first service, determine the second service node as a preliminarily selected abnormal node.

[0044] Optionally, the processing module is further used for:

[0045] Before the monitoring detects that the first service node generates an abnormality in processing the first service, the third service nodes processing the third service are determined through the service processing flow of the third service, and the indicator curve associated with the third service among the indicator curves of each third service node is determined; the third service is any one of the services;

[0046] A comparison relationship among a service identifier of the third service, each third service node, and an indicator curve associated with the third service in each third service node is generated.

[0047] Optionally, the processing module is specifically used for:

[0048] Determine a first ranking of the first abnormal start time, the first abnormal peak time, and the indicator fluctuation values ​​at each time in the abnormal period in the first indicator curve;

[0049] Determine a second ranking of the second abnormal start time, the second abnormal peak time, and the indicator fluctuation values ​​at each time in the abnormal period in the second indicator curve;

[0050] According to a first matching degree between the first abnormality start time and the second abnormality start time, a second matching degree between the first abnormality peak time and the second abnormality peak time, and a third matching degree between the first sorting and the second sorting, it is determined whether the matching degree between the first indicator curve and the second indicator curve is greater than a set threshold.

[0051] Optionally, the processing module is specifically used for:

[0052] For the same indicator where an abnormality occurs, determining a first change amount of the indicator by the first service node and a second change amount of the indicator by the preliminarily selected abnormal node;

[0053] If the relationship between the first change amount and the second change amount does not meet the set condition, excluding the initially selected abnormal node;

[0054] A root cause node causing the exception is determined from the preliminarily selected exception nodes and the first service node after the exclusion operation.

[0055] Optionally, the processing module is specifically used for:

[0056] For the indicator corresponding to the first indicator curve, determine the influencing party on the indicator when processing the first business from the preliminarily selected abnormal nodes and the first business node, and determine the influencing party as the root cause node causing the abnormality.

[0057] Optionally, the processing module is further used for:

[0058] Determining a first attribute associated with the indicator corresponding to the first indicator curve and a second attribute associated with the indicator corresponding to the second indicator curve;

[0059] The same attributes of the first attributes and the second attributes are displayed.

[0060] In a third aspect, an embodiment of the present invention provides a computer controlled device, including:

[0061] A memory for storing program instructions;

[0062] The processor is used to call the program instructions stored in the memory and execute the method described in the first aspect according to the obtained program.

[0063] In a fourth aspect, an embodiment of the present invention provides a computer-readable non-volatile storage medium, comprising computer-readable instructions. When a computer reads and executes the computer-readable instructions, the computer executes the method described in the first aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0065] Figure 1 A schematic diagram of a system architecture provided for an embodiment of the present invention;

[0066] Figure 2 A schematic diagram of a root cause analysis method provided by an embodiment of the present invention;

[0067] Figure 3 A schematic diagram of a root cause analysis method provided by an embodiment of the present invention;

[0068] Figure 4 A schematic diagram of a root cause analysis method provided by an embodiment of the present invention;

[0069] Figure 5 A schematic diagram of a root cause analysis method provided by an embodiment of the present invention;

[0070] Figure 6 A schematic diagram of a root cause analysis method provided by an embodiment of the present invention;

[0071] Figure 7 A schematic diagram of the structure of a root cause analysis device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0072] In order to better understand the above-mentioned technical scheme, the above-mentioned technical scheme will be described in detail below in conjunction with the accompanying drawings and specific implementation methods of the specification. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical scheme of the present invention, rather than limitations on the technical scheme of the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments may be combined with each other.

[0073] First, some terms in the present application are explained so as to be understood by those skilled in the art.

[0074] Business indicators: indicators related to business behavior in the monitoring system, such as transaction volume, success rate, time consumption, success volume, etc.

[0075] Business exception: An exception occurs in a certain application scenario of a business product, such as WeChat payment failure.

[0076] Transaction flow: A number used to mark a transaction, which can circulate in multiple subsystems used to process the transaction.

[0077] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings.

[0078] Figure 1 This is a schematic diagram of a system architecture applicable to the embodiment of the present invention, which can be figuratively called a business processing tree. The present invention is based on the premise that the business indicators of the nodes on the tree are fully reported. Among them, the construction method of the business processing tree is based on a large amount of transaction flow data, thereby constructing a system call tree diagram of a business scenario.

[0079] It should be noted that in this tree diagram, each business system can independently report the corresponding business indicators to the tree nodes, and bind relevant attributes to each indicator, such as: corresponding external partners, related institutions and banks, corresponding backend interfaces and services, dimensions and categories, etc.

[0080] Further, after completing the above basic work preparations, you can choose to perform curve anomaly detection on the business indicators of each node in the tree, or perform curve anomaly detection on the business indicators of one of the nodes in the tree, which can be a root node or a child node, and this application does not make specific restrictions on this. At this time, a curve anomaly detection algorithm can be used for detection. In order to improve the accuracy and intelligence of the analysis, this application preferably performs curve anomaly detection on the business indicators of each node in the tree.

[0081] In one possible solution, for example, 3-sigma is used for testing: first assume that a set of test data contains only random errors, calculate and process them to obtain the standard deviation, and determine an interval according to a certain probability. It is considered that any error exceeding this interval is not a random error but a gross error, and the data containing this error is eliminated.

[0082] In the embodiment of the present application, when a business anomaly occurs, it can be found that the indicators between the subsystems in the tree have some abnormal correlation, for example, each subsystem will rise or fall at the same time near the abnormal point. Based on this feature, the embodiment of the present application adopts a method of judging the similarity of different curves to perform root cause analysis.

[0083] In order to better explain the above embodiments, Figure 2 The flow of a root cause analysis method is exemplarily shown, and the flow can be executed by a root cause analysis device.

[0084] like Figure 2 As shown in the figure, the process specifically includes:

[0085] Step 201: after monitoring that a first service node generates an abnormality in processing a first service, a first indicator curve in which the abnormality occurs in the first service node is obtained.

[0086] It should be noted that the present application obtains the first indicator curve that sends the anomaly after the anomaly detection finds the anomaly. In one possible design scheme, when one of the multiple indicator curves on a node does not meet the preset conditions at a certain moment, the alarm will generate an alarm to indicate the anomaly. In another possible design scheme, the system maintenance personnel detects the anomaly based on the preset core indicators. When the corresponding indicator curve does not meet the preset conditions, the system maintenance personnel determines that the indicator curve has an anomaly.

[0087] Step 202: determine a second service node corresponding to the first service.

[0088] It should be noted that the second service node is any service node among the service nodes processing the first service except the first service node.

[0089] Step 203: determine a preliminarily selected abnormal node from each second service node.

[0090] It should be noted that the initially selected abnormal nodes have a second indicator curve that matches the first indicator curve.

[0091] Step 204: determine the root cause node causing the exception from the first service node and the preliminarily selected exception nodes.

[0092] In the embodiment of the present application, because the indicators of the entry-side nodes are generally easier to manually mark the product and business affiliation, for example: a certain indicator corresponds to the number of visits to WeChat, and the sub-nodes on the non-entry side are generally shared by multiple products or businesses, for example: a deposit subsystem may have many business accesses at different entrances.

[0093] From the above content, it can be seen that the indicators corresponding to non-entry nodes are difficult to bind directly to products or businesses. If the correspondence between non-entry side node indicators and businesses can be found, then when the indicators of a certain business are abnormal, all indicator sets associated with the scenario can be quickly found, rather than the full indicator set (that is, the indicator set that is not related to the business), thereby achieving the purpose of narrowing the matching range.

[0094] Therefore, in step 202, in order to determine the second service node corresponding to the first service, before monitoring in step 201 that the first service node processes the first service abnormally, a comparison relationship between the indicator curve and the service node is established in the following manner:

[0095] Specifically, first, the third service nodes for processing the third service are determined through the service processing flow of the third service, and the indicator curve associated with the third service among the indicator curves of each third service node is determined.

[0096] It should be noted that the third service is any one of the services.

[0097] Finally, a comparison relationship among the service identifier of the third service, each third service node, and an indicator curve associated with the third service in each third service node is generated.

[0098] It can be seen from the above content that, through the service processing flow of the third service, the nodes through which the third service passes and the indicator curves on the nodes in which the third service participates can be determined.

[0099] For example, a subsystem X has 10,000 log data containing transaction flows at a certain moment. These 10,000 logs can be used to calculate 100 indicator curves based on the calculation rules configured by the user, such as log filtering rules. If a log meets a certain rule, it is determined to belong to a certain curve. For example, if the log contains "success", it means that it is related to the success rate indicator.

[0100] Based on the above content, select a transaction flow at the root node of the tree, and examine which child nodes downstream this transaction flow has appeared in, and which indicators it has participated in. Then, as long as this transaction flow is bound to a business, all indicators corresponding to the child nodes can be bound to the business. The corresponding relationship between indicators and businesses obtained by this method is a one-to-many relationship, that is, one indicator can participate in the formation of multiple businesses.

[0101] Take a simple example, such as Figure 3 As shown in the figure, transaction A is a WeChat access that passes through the three nodes "1", "4" and "8" in sequence and participates in the success rate indicator on node 1. Transaction B is a QQ access that passes through the two nodes "1" and "5" in sequence and also participates in the success rate indicator on node 1. Then the indicator curve corresponding to the success rate on node "1" will be associated with the two businesses of WeChat access and QQ access.

[0102] Furthermore, since a transaction can only reflect part of the situation of a certain business, that is, a transaction may not necessarily flow to all child nodes and all indicators, the above solution is implemented in batches.

[0103] In a possible solution design, a batch of transaction flow numbers corresponding to the entry-side indicators can be collected. Preferably, the flow of these indicators in each node in the tree is counted offline, and the composition of the transaction flow and the indicator curve in the node is analyzed.

[0104] The above scheme avoids the situation where some originally unrelated curves happen to show abnormal changes at the same time point due to the scope of similarity judgment being too large (when the sample reaches a certain amount, there will always be some curves that happen to show normal abnormal changes at a certain moment). Through the above offline analysis, it is equivalent to finding which indicators in the tree are related to each indicator of the entrance and which are not related through coloring. When making similarity judgments, only the sets with related ones are searched. In this way, the sets for judging similarity are greatly reduced, and the accuracy is improved.

[0105] In step 203 of the embodiment of the present application, specifically, for each second business node, an indicator curve associated with the first business among the indicator curves of the second business node is determined; if there is a second indicator curve matching the first indicator curve among the indicator curves associated with the first business, the second business node is determined to be a preliminarily selected abnormal node.

[0106] Furthermore, in order to implement the above determination method, it is necessary to find curves in each node of the tree that have similar index changes to the abnormal index curve. In other words, the embodiment of the present application provides a method to determine whether the trends of various curves in the abnormal time period are similar.

[0107] It should be noted that the embodiment of the present application does not focus on whether a curve is similar or has the same trend as a whole, but only focuses on whether different curves have the same fluctuation near the time point when the indicator is abnormal.

[0108] Specifically, the second indicator curve that matches the first indicator curve is determined in the following manner. The specific process is as follows: Figure 4 As shown, including:

[0109] Step 401, determining a first ranking of the first abnormal start time, the first abnormal peak time, and the index fluctuation values ​​at each time in the abnormal period in the first index curve;

[0110] Step 402, determining a second ranking of the second abnormal start time, the second abnormal peak time, and the indicator fluctuation values ​​at each time in the abnormal period in the second indicator curve;

[0111] Step 403, determining whether the matching degree between the first indicator curve and the second indicator curve is greater than a set threshold based on a first matching degree between the first abnormal start time and the second abnormal start time, a second matching degree between the first abnormal peak time and the second abnormal peak time, and a third matching degree between the first sorting and the second sorting.

[0112] In the embodiment of the present application, three features of the curve are selected, including the abnormal start time, the abnormal peak time, and the ranking of the index fluctuation values ​​at each time in the abnormal period, and these three features are used to judge the similarity. It should be noted that other features such as the abnormal end time, the abnormal duration, and the difference between the maximum and minimum values ​​of the index in the abnormal time period can also be selected to judge the similarity. This application does not make specific restrictions on this. The following describes the acquisition methods of these three features:

[0113] Abnormal start time: Use curve anomaly detection algorithm such as 3-Sigma to find the start time of the anomaly.

[0114] Abnormal peak moment: Find the time point corresponding to the lowest transaction volume, success rate, or the highest time consumption within the time period.

[0115] Sorting of indicator fluctuation values: For each point in the curve, find the average value corresponding to the first N points of the point, and subtract the previous average value from the value to get the difference value of the point, i.e., the fluctuation value. Find the fluctuation values ​​of all points X hours before the abnormal time point of the curve, and sort the fluctuation values ​​to get the ranking of the fluctuation values ​​corresponding to the current curve.

[0116] For example, the sorting of the above indicator fluctuation values ​​is carried out. For example, find the point at 18:20 and obtain the average value of the points contained in the 20 minutes before this point. For example, collect a point every one minute, and get the indicator values ​​of 20 points. Take the average value and then sort the average values ​​from large to small.

[0117] Furthermore, after completing the collection of the above three features, the score can be given by the degree of match between the features of a certain curve and the features of the abnormal indicator curve. In a possible solution design, 3 points are awarded for completely identical start times, and 1 point is deducted for every 1 minute difference. 3 points are awarded for completely identical peak times, and 1 point is deducted for every 1 minute difference. 3 points are awarded for completely identical ranking of fluctuation values, and 1 point is deducted for every 1 minute difference. If a class value is negative, it is directly excluded.

[0118] From the above content, it can be seen that through the weighted summation of the above scores, the similarity score between a certain curve and the abnormal indicator curve can be obtained. When it is higher than a certain threshold, it is determined as a preliminary abnormal node.

[0119] It can be seen from the above content that the indicator curve associated with the first business is determined from the indicator curves of the second business node corresponding to the first business, and the indicator curve associated with the first business is matched; in actual business, a business node may involve multiple businesses, and a business node may be set with many indicator curves based on various requirements; in this solution, the efficiency of root cause analysis can be improved by narrowing the scope of business nodes and the scope of indicator curves; at the same time, since irrelevant curves are excluded, the matching degree of some curves with the first indicator curve due to fluctuations is avoided, thereby improving the accuracy of root cause analysis.

[0120] However, some misjudgments may still be found through the above-mentioned curve similarity judgment method. That is to say, after verification, the results of the root cause analysis are found not to be the real root causes. The reason for the misjudgment may be that some curves that are judged to be similar anomalies, from a logical level, also found unreasonable data, such as: the entry time increased by 1000ms, the core side only increased by 20ms, that is, the increase of 20ms on the core side led to an increase of 1000ms on the entry side, which is obviously unreasonable. Based on this, in step 204 of the embodiment of the present application, specifically through the following Figure 5The method flow shown excludes a portion of nodes from the initially selected abnormal nodes.

[0121] The specific process is as follows:

[0122] Step 501, for the same indicator where an abnormality occurs, determine a first change amount of the indicator of the first business node and a second change amount of the indicator of the preliminarily selected abnormal node.

[0123] Step 502: If the relationship between the first variation and the second variation does not satisfy the set condition, the initially selected abnormal node is excluded.

[0124] Step 503: determine the root cause node causing the exception from the initially selected abnormal nodes and the first service node after the elimination operation.

[0125] The above scheme further improves the accuracy of root cause analysis by judging that the rationality of data correlation is low.

[0126] For example, first analyze all the curves with similar anomalies through the above content. When the success rate on the entry side decreases, the success amount and the total amount used to calculate the current success rate can be found, that is, success rate = success amount / total amount, and the failure amount can be further obtained, that is, failure amount = total amount-success amount. Through this method, the failure amount corresponding to all similar curves can be found. If the failure amount of a certain node is too low compared to the failure amount of the node where the abnormality occurs, it means that it is impossible for the failure of this node to cause a large number of failures of the node where the abnormality occurs, and the possibility of this similar node being the root cause is ruled out; when the time consumption increases, similar processing can also be done, and this application will not go into details here.

[0127] For example, if the anomaly occurs at the entry node, the transaction volume of the entry node drops by 1,000, while the transaction volume of the primary abnormal node only drops by 10, it means that the primary abnormal node is not the main cause and should be excluded.

[0128] The above method can eliminate some curves that were originally judged to be similar but unreasonable, thereby improving the accuracy of the judgment.

[0129] Furthermore, after determining the preliminary abnormal node, the embodiment of the present application determines the influencing party on the indicator when processing the first business from the preliminary abnormal node and the first business node for the indicator corresponding to the first indicator curve, and determines the influencing party as the root cause node that causes the abnormality.

[0130] Specifically, if the transaction volume on the entry side decreases, and the transaction volume of some subsequent subsystems also decreases, it can be determined that the decrease in traffic on the entry side has led to a decrease in the back-end transaction volume indicator, and the root cause is the entry node.

[0131] If a sudden drop in success rate or a sudden increase in time consumption occurs on the entry side, and some subsequent subsystems also experience abnormalities in the same business indicators, it can be determined that the decrease in success rate or increase in time consumption on the backend has led to a decrease in success rate or an increase in time consumption on the entry side. The root cause is the last abnormal tree node, that is, the decrease in success rate or increase in time consumption on the subsequent nodes will definitely be fed back to the previous nodes.

[0132] The above scheme determines the final root cause node according to the dependency relationship between the initially selected abnormal node and the abnormal node, thereby reducing the possibility of misjudgment and improving the accuracy of root cause analysis.

[0133] Furthermore, after the RCA judgment of the above solution, the root cause node and the root cause index can be obtained. However, this information is not easy to read for people other than system developers. The embodiment of the present application further improves the readability of the root cause analysis conclusion through the following method. The specific process is as follows: Figure 6 shown.

[0134] Step 601: Determine a first attribute associated with an indicator corresponding to a first indicator curve and a second attribute associated with an indicator corresponding to a second indicator curve.

[0135] Step 602: display the same attributes of the first attribute and the second attribute.

[0136] Specifically, find all indicators with similar changes and the attributes associated with the indicators, and find out the common points. For example, the indicators are all associated with an external partner, a bank, etc. Through the statistics and output of this information, the anomalies can be confined to a smaller range or category.

[0137] Furthermore, in the method for searching for associated attributes in the embodiment of the present application, in one possible implementation, when a user reports an indicator, the related attributes are bound in the reported data, and in another possible implementation, the related attributes are edited in the management console after the report is completed. In addition, the name of the indicator may also be fuzzily matched to the name of the bank or institution. For example, an indicator name may be: CCB interface call success rate. By binding the indicator to a bank or bank abbreviation dictionary, the related bank can be extracted from the name as: CCB.

[0138] In a possible implementation, for the exception of a reduced success rate of the system, relevant information of the failed transaction logs corresponding to all indicators can be stored in advance to form an error cause data information database. When the success rate is reduced, the relevant logs of all indicators that cause the reduction in success rate can be found in the information database, and relevant information can be extracted from the logs, for example, Chinese characters involved in the logs, specific categories such as connection timeout, database access exceptions, etc. Through the statistics and output of relevant logs, users can also better understand the cause of the problem.

[0139] The above solution can confine the anomaly to a smaller range or category by finding all indicators with similar changes and the attributes associated with the indicators and performing statistics and output. This improves the readability of the root cause analysis results. Looking at the problem from a macro perspective can more accurately and comprehensively discover the root cause of the problem, and can better find the classification or direction of the problem. Through this classification or direction, combined with micro analysis, a better conclusion of the root cause analysis can be drawn.

[0140] Based on the same technical concept, Figure 7 The structure of a root cause analysis device provided by an embodiment of the present invention is exemplarily shown, and the device can execute the process of the root cause analysis method.

[0141] like Figure 7 As shown, the device may include:

[0142] An acquisition module 701 is used to acquire a first indicator curve in which an exception occurs in a first service node after monitoring that an exception occurs in a first service node processing a first service;

[0143] A processing module 702 is used to determine a second service node corresponding to the first service, where the second service node is any service node among the service nodes processing the first service except the first service node;

[0144] The processing module 702 is further configured to determine a preliminarily selected abnormal node from each second service node, wherein the preliminarily selected abnormal node has a second indicator curve matching the first indicator curve;

[0145] The processing module 702 is further configured to determine a root cause node causing the exception from the first service node and the preliminarily selected exception nodes.

[0146] Optionally, the processing module 702 is specifically configured to:

[0147] For each second service node, determine the indicator curve associated with the first service among the indicator curves of the second service node; if there is a second indicator curve matching the first indicator curve among the indicator curves associated with the first service, determine the second service node as a preliminarily selected abnormal node.

[0148] Optionally, the processing module 702 is further used for:

[0149] Before the monitoring detects that the first service node generates an abnormality in processing the first service, the third service nodes processing the third service are determined through the service processing flow of the third service, and the indicator curve associated with the third service among the indicator curves of each third service node is determined; the third service is any one of the services;

[0150] A comparison relationship among a service identifier of the third service, each third service node, and an indicator curve associated with the third service in each third service node is generated.

[0151] Optionally, the processing module 702 is specifically configured to:

[0152] Determine a first ranking of the first abnormal start time, the first abnormal peak time, and the indicator fluctuation values ​​at each time in the abnormal period in the first indicator curve;

[0153] Determine a second ranking of the second abnormal start time, the second abnormal peak time, and the indicator fluctuation values ​​at each time in the abnormal period in the second indicator curve;

[0154] According to a first matching degree between the first abnormality start time and the second abnormality start time, a second matching degree between the first abnormality peak time and the second abnormality peak time, and a third matching degree between the first sorting and the second sorting, it is determined whether the matching degree between the first indicator curve and the second indicator curve is greater than a set threshold.

[0155] Optionally, the processing module 702 is specifically configured to:

[0156] For the same indicator where an abnormality occurs, determining a first change amount of the indicator by the first service node and a second change amount of the indicator by the preliminarily selected abnormal node;

[0157] If the relationship between the first change amount and the second change amount does not meet the set condition, excluding the initially selected abnormal node;

[0158] A root cause node causing the exception is determined from the preliminarily selected exception nodes and the first service node after the exclusion operation.

[0159] Optionally, the processing module 702 is specifically configured to:

[0160] For the indicator corresponding to the first indicator curve, determine the influencing party on the indicator when processing the first business from the preliminarily selected abnormal nodes and the first business node, and determine the influencing party as the root cause node causing the abnormality.

[0161] Optionally, it is characterized in that the processing module 702 is further used for:

[0162] Determining a first attribute associated with the indicator corresponding to the first indicator curve and a second attribute associated with the indicator corresponding to the second indicator curve;

[0163] The same attributes of the first attribute and the second attribute are displayed. Based on the same technical concept, an embodiment of the present invention also provides a computing controlled device, including:

[0164] A memory for storing program instructions;

[0165] The processor is used to call the program instructions stored in the memory and execute the above-mentioned method for screening machine learning models according to the obtained program.

[0166] Based on the same technical concept, an embodiment of the present invention also provides a computer-readable non-volatile storage medium, including computer-readable instructions. When a computer reads and executes the computer-readable instructions, the computer executes the above-mentioned method for screening machine learning models.

[0167] Finally, it should be noted that: It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, optical storage, etc.) containing computer-usable program code.

[0168] The present invention is described with reference to the flowcharts and / or block diagrams of the method, controlled device (system), and computer program product according to the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing controlled device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing controlled device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 Process or multiple and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0169] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing controlled device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0170] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A root cause analysis method, characterized in that: include: After monitoring that the first service node generates an abnormality in processing the first service, obtaining a first indicator curve in which the abnormality occurs in the first service node; Determine a second service node corresponding to the first service, where the second service node is any service node among the service nodes processing the first service except the first service node; For each second service node, determining a second indicator curve associated with the first service among the indicator curves of the second service node; Determine a first ranking of the first abnormal start time, the first abnormal peak time, and the indicator fluctuation values ​​at each time in the abnormal period in the first indicator curve; Determine a second ranking of the second abnormal start time, the second abnormal peak time, and the indicator fluctuation values ​​at each time in the abnormal period in the second indicator curve; Determining whether the matching degree between the first indicator curve and the second indicator curve is greater than a set threshold according to a first matching degree between the first abnormal start time and the second abnormal start time, a second matching degree between the first abnormal peak time and the second abnormal peak time, and a third matching degree between the first sorting and the second sorting; If the matching degree between the first indicator curve and the second indicator curve is greater than a set threshold, determining the second service node as a preliminary abnormal node; A root cause node causing the exception is determined from the first service node and the preliminarily selected exception nodes.

2. The method according to claim 1, characterized in that Before monitoring that the first service node generates an abnormality in processing the first service, the method further includes: Determine, through a service processing flow of the third service, each third service node for processing the third service, and determine, among each indicator curve of each third service node, an indicator curve associated with the third service; the third service is any one of the services; A comparison relationship among a service identifier of the third service, each third service node, and an indicator curve associated with the third service in each third service node is generated.

3. The method according to claim 1, characterized in that: The step of determining the root cause node causing the exception from the first service node and the preliminarily selected exception node includes: For the same indicator where an abnormality occurs, determining a first change amount of the indicator by the first service node and a second change amount of the indicator by the preliminarily selected abnormal node; If the relationship between the first change amount and the second change amount does not meet the set condition, excluding the initially selected abnormal node; A root cause node causing the exception is determined from the preliminarily selected exception nodes and the first service node after the exclusion operation.

4. The method according to claim 3, characterized in that: Determining the root cause node causing the abnormality from the preliminarily selected abnormal nodes and the first service node after the exclusion operation includes: For the indicator corresponding to the first indicator curve, determine the influencing party on the indicator when processing the first business from the preliminarily selected abnormal nodes and the first business node, and determine the influencing party as the root cause node causing the abnormality.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Determining a first attribute associated with the indicator corresponding to the first indicator curve and a second attribute associated with the indicator corresponding to the second indicator curve; The same attributes of the first attributes and the second attributes are displayed.

6. A root cause analysis device, characterized in that: The device comprises: An acquisition module, configured to acquire a first indicator curve in which the first service node has an abnormality after monitoring that the first service node has an abnormality in processing the first service; A processing module, used to determine a second service node corresponding to the first service, where the second service node is any service node among the service nodes processing the first service except the first service node; The processing module is further used to determine, for each second business node, a second indicator curve associated with the first business among the indicator curves of the second business node; determine a first ranking of the indicator fluctuation values ​​of the first abnormal start time, the first abnormal peak time, and each time in the abnormal period in the first indicator curve; determine a second ranking of the indicator fluctuation values ​​of the second abnormal start time, the second abnormal peak time, and each time in the abnormal period in the second indicator curve; determine whether the matching degree between the first indicator curve and the second indicator curve is greater than a set threshold according to a first matching degree between the first abnormal start time and the second abnormal start time, a second matching degree between the first abnormal peak time and the second abnormal peak time, and a third matching degree between the first ranking and the second ranking; if the matching degree between the first indicator curve and the second indicator curve is greater than the set threshold, determine that the second business node is a primary abnormal node; The processing module is further used to determine the root cause node causing the exception from the first service node and the preliminarily selected exception nodes.

7. A computing device, characterized in that include: A memory for storing program instructions; A processor, configured to call the program instructions stored in the memory, and execute the method according to any one of claims 1 to 5 according to the obtained program.

8. A computer-readable non-volatile storage medium, characterized in that: The method comprises computer-readable instructions, and when a computer reads and executes the computer-readable instructions, the computer is caused to execute the method according to any one of claims 1 to 5.

Images