A root cause positioning method and system based on full-link topology and multi-dimensional analysis

Abstract

The application provides a root cause positioning method and system based on full-link topology and multi-dimensional analysis, relates to the technical field of software monitoring and fault troubleshooting, and comprises the following steps: establishing a cross-application horizontal positioning method, wherein the cross-application horizontal positioning method comprises transaction time consumption type abnormality positioning and transaction failure type abnormality positioning; and establishing a single-application vertical positioning method, wherein the single-application vertical positioning method comprises application multi-dimensional drilling analysis and application associated resource drilling analysis. The application shortens the fault troubleshooting time, reduces the operation and maintenance cost, and improves the availability and stability of the system.

Description

Technical Field

[0001] This invention relates to the field of software monitoring and troubleshooting technology, and in particular to a root cause localization method and system based on full-link topology and multi-dimensional analysis. Background Technology

[0002] Traditional application troubleshooting methods mainly rely on manual experience, gradually identifying the cause of failures through log analysis, indicator monitoring, and other methods. However, these methods have the following shortcomings:

[0003] 1. Low troubleshooting efficiency: In the face of complex distributed systems, the inter-application call relationships are complicated, and manual troubleshooting is difficult to quickly clarify the call chain and locate the faulty link, which is time-consuming and laborious.

[0004] 2. Insufficient accuracy: Relying solely on human experience makes it difficult to comprehensively and accurately analyze massive amounts of monitoring data, which can easily lead to misjudgments or omissions of potential fault points, resulting in the inability to accurately locate the root cause of the fault.

[0005] 3. Limited data processing capabilities: Traditional methods struggle to perform effective correlation analysis and mining on large-scale monitoring data, failing to fully utilize the potential information within the data to assist in fault location.

[0006] 4. Lack of intelligence and automation: Traditional investigation methods rely heavily on manual intervention and lack intelligent analysis models and automated means, making it difficult to adapt to the rapid iteration and complex and ever-changing characteristics of modern distributed systems.

[0007] Therefore, it is necessary to provide a root cause localization method and system based on full-link topology and multi-dimensional analysis to efficiently and accurately locate the root cause of application failures, shorten troubleshooting time, reduce operation and maintenance costs, and improve system availability and stability. Summary of the Invention

[0008] The purpose of this invention is to provide a root cause localization method and system based on full-link topology and multi-dimensional analysis, so as to efficiently and accurately locate the root cause of application failure, shorten the troubleshooting time, reduce operation and maintenance costs, and improve the availability and stability of the system.

[0009] To address the problems existing in the prior art, this invention provides a root cause localization method based on end-to-end topology and multi-dimensional analysis, comprising the following steps:

[0010] A cross-application horizontal positioning method is established, which includes transaction time-related anomaly positioning and transaction failure anomaly positioning. The transaction time-related anomaly positioning is completed by stripping downstream time-consuming processes, and the transaction failure anomaly positioning is completed by locating the deepest and most frequently hit failure nodes.

[0011] A single-application vertical positioning method is established, which includes application multidimensional drill-down analysis and application-related resource drill-down analysis. The application multidimensional drill-down analysis distinguishes between normal and abnormal sample data, calculates the differences between them in data distribution, feature values, or statistical characteristics, and uses statistical methods to quantify the differences to locate the root cause dimension of the fault. The application-related resource drill-down analysis sets clear trigger conditions for specific indicators based on rules, filters abnormal indicators of the resource model through trigger conditions, and sorts them based on the strength of the dependency model between abnormal indicators and applications. The abnormal indicators of the resource model are displayed and statistically analyzed based on the sorting to recommend and locate the fault.

[0012] Optionally, in the root cause localization method based on end-to-end topology and multi-dimensional analysis, the method for locating transaction time-related anomalies is as follows:

[0013] S11: Obtain sample data and preprocess the sample data, which includes the average transaction processing time.

[0014] S12: Query abnormal samples based on the application PASO, the three minutes before and the two minutes after the abnormal time point and / or the abnormal time consumption value greater than or equal to the alarm content, where application PASO is an abbreviation for query tag;

[0015] S13: Based on the queried anomaly samples, locate the anomaly and construct the full-link transaction topology. Subtract the total time of all downstream calls from the total time of the ingress request to obtain the actual time of the application's own business logic, as follows:

[0016] S131: Calculate the application's average execution time using the following formula:

[0017] Among them, T self To determine the actual time taken for a single transaction based on its own business logic, T total T represents the total processing time of the application. downstream_i Let be the call time of the i-th downstream application, i = 1, 2, ..., n, where n is the number of downstream calls. This is the sum of the times taken by all downstream calls;

[0018] ;in, The average time taken to apply the method is given, where m is the total number of samples, j is the sample index, and n is the average time taken to apply the method. j Let J be the number of downstream calls involved in the j-th sample.

[0019] S132: Calculate the absolute deviation of the application's own time consumption. Calculate the deviation between the average time consumption of each application in the current alarm period and the average time consumption in the alarm period one week ago in the sampled logs. The formula is as follows:

[0020] Absolute deviation = ;

[0021] in, The average time consumed by the application within the current alarm period. The average latency is the historical baseline for the same alarm period one week ago. and Calculated based on the application's own average time consumption formula;

[0022] S133: Calculate the downstream application anomaly impact score using the following formula:

[0023] Anomaly impact score = absolute deviation × transaction ratio, where transaction ratio = current downstream application transaction volume / total transaction volume across the entire chain;

[0024] S134: Influence score ranking and recommendation positioning through downstream application anomalies.

[0025] Optionally, in the root cause localization method based on end-to-end topology and multi-dimensional analysis, the method for locating transaction failure anomalies is as follows:

[0026] S21: When abnormal alarms occur in the technical success rate indicator, technical failure rate indicator, and business acceptance rate indicator, S22-S24 are triggered.

[0027] S22: Construct a full-link transaction topology based on failure-type anomaly localization to complete data preprocessing;

[0028] S23: Based on anomalies in technical success rate indicators and / or technical failure rate indicators, combined with the application PASO, the three minutes before and the two minutes after the abnormal time point, and the transaction technical status, query abnormal samples for failure; and / or, based on anomalies in business acceptance rate, combined with the application PASO, the three minutes before and the two minutes after the abnormal time point, the transaction technical status is failed, and the transaction business status is failed, query abnormal samples for failure, where application PASO is an abbreviation for query tag;

[0029] S24: Locate the anomaly based on the retrieved samples, as follows:

[0030] S241: Filter out all nodes in the abnormal samples, extract the deepest node on all call paths, and use the deepest failure node as the candidate node for the cause of failure.

[0031] S242: Count the number of times each node records the reason for failure in the transaction data, and select the candidate node with the most failures in the leaf nodes as the root cause node of the anomaly.

[0032] Optionally, in the root cause localization method based on end-to-end topology and multi-dimensional analysis, the multi-dimensional drill-down analysis method is applied as follows:

[0033] S31: Define the scope of the analysis data as the application transaction detail log;

[0034] S32: Define the trigger condition as all transaction alerts;

[0035] S33: Distinguish between normal sample data and abnormal sample data, calculate the differences between the two in terms of data distribution, feature values ​​or statistical characteristics, and use statistical methods to quantify the differences to locate the root cause dimension of the fault.

[0036] Optionally, in the root cause localization method based on end-to-end topology and multi-dimensional analysis, statistical methods are used to quantify differences to locate the root cause dimensions of the fault, as follows:

[0037] Configure the transaction statistics dimensions for deviation, including microservice identifier, transaction type, logical data center identifier, IP address, cluster region identifier, region identifier, combined dimensions including transaction type, technical error code and technical error information, and combined dimensions including transaction type, business error code and business error information;

[0038] Calculate the characteristics of each dimension value, including the abnormal traffic percentage and the normal traffic percentage of the current dimension value;

[0039] The formula for calculating the percentage of abnormal traffic for the current dimension value is:

[0040] P （维度值i） This represents the percentage of abnormal traffic for the current dimension value.

[0041] The formula for calculating the normal traffic percentage of the current dimension value is:

[0042] Q （维度值i） This represents the normal traffic percentage for the current dimension value.

[0043] The JS divergence for each dimension is calculated using the following formula:

[0044] Where P and Q represent the probability distributions of abnormal and normal samples in any dimension, respectively. M represents the average distribution of P and Q, KL(P||M) is the KL divergence between P and M, and KL(Q||M) is the KL divergence between Q and M;

[0045] Filtering is performed when the JS divergence is less than 0.1%, and the results are sorted by JS divergence.

[0046] The top 3 values ​​based on the JS divergence, from largest to smallest, are used as the root causes for multidimensional drill-down analysis.

[0047] Optionally, in the root cause localization method based on end-to-end topology and multi-dimensional analysis, the associated resource drill-down analysis method is applied as follows:

[0048] S41: Set explicit trigger conditions for specific indicators based on rules. Specific indicators include status indicators, saturation indicators, and non-capacity indicators. The trigger conditions are as follows:

[0049] Status indicators:

[0050] Simultaneity: Status indicators show a change in status due to anomalies within ten minutes before and after an alarm.

[0051] Anomalies: An alarm is triggered when a status metric exceeds a specific threshold;

[0052] Saturation index:

[0053] Simultaneity: It is required that there is a sudden increase in the saturation index at the time of the alarm, and the saturation index increases significantly by 30% year-on-year between the ten minutes before the alarm and the ten minutes after the alarm time.

[0054] Anomalies: An alarm is triggered when the saturation index exceeds a specific threshold;

[0055] Non-capacity metrics:

[0056] Simultaneity: It requires that there is a sudden increase in non-capacity indicators at the time of alarm, and that the non-capacity indicators increase significantly by 30% year-on-year between the ten minutes before the alarm and the ten minutes after the alarm time.

[0057] Anomalies: Detecting sudden increases compared to historical data;

[0058] S42: Filter abnormal indicators of the resource model by trigger conditions;

[0059] S43: Sort based on strong and weak dependency models of abnormal indicators and application associations, and combine the sorting to display and statistically analyze abnormal indicators of resource models for recommendation positioning.

[0060] Optionally, in the root cause localization method based on end-to-end topology and multi-dimensional analysis,

[0061] Status indicators include status 0 and status 1, which are used to display the database connection status;

[0062] Saturation metrics include CPU utilization, memory utilization, or connection pool utilization;

[0063] Non-capacity metrics include absolute values, which include requests per second and queue depth.

[0064] The present invention also provides a root cause localization system based on end-to-end topology and multi-dimensional analysis, and the root cause localization system is constructed using the root cause localization method.

[0065] Compared with the prior art, the present invention has the following advantages:

[0066] (1) This invention aims to provide a root cause localization method and system based on end-to-end topology and multi-dimensional analysis, so as to shorten the fault diagnosis time, reduce operation and maintenance costs, improve the availability and stability of the system, and achieve:

[0067] 1) Precise location of cross-application anomaly propagation paths;

[0068] 2) Anomaly localization based on multi-dimensional features within a single application;

[0069] 3) Collaborative analysis of anomalies in the resource layer and application layer.

[0070] (2) Improve fault location efficiency: Through automated and intelligent analysis algorithms, the entire transaction topology can be quickly constructed, abnormal samples can be accurately screened, and the root cause of the fault can be quickly located based on the deviation of multiple dimensions. Compared with the traditional manual troubleshooting method, the fault troubleshooting time is greatly shortened and the operation and maintenance efficiency is improved.

[0071] (3) Improve the accuracy of fault location: Multi-dimensional data analysis can more comprehensively and deeply explore the potential information in the data, accurately identify the factors that have a significant impact on the fault, reduce the risk of misjudgment and omission, and more accurately locate the root cause of the fault compared with traditional methods, providing a more reliable basis for fault repair.

[0072] (4) Enhance system stability and availability: timely and accurate fault location and repair reduces the impact time of faults on system operation, improves the overall stability and availability of the system, and ensures business continuity and user satisfaction.

[0073] (5) Reduced operation and maintenance costs: It reduces the workload and time of manual troubleshooting, reduces the workload of operation and maintenance personnel and their dependence on experience, and avoids business losses and potential cost increases caused by long-term unresolved faults, bringing significant cost benefits to the enterprise's IT operation and maintenance. Attached Figure Description

[0074] Figure 1 This is a flowchart illustrating the root cause localization method provided in an embodiment of the present invention. Detailed Implementation

[0075] The specific embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. The advantages and features of the present invention will become clearer from the following description. It should be noted that the accompanying drawings are all in a very simplified form and use non-precise proportions, and are only used to facilitate and clarify the illustration of the embodiments of the present invention.

[0076] In the following, if the methods described herein include a series of steps, the order of these steps presented herein is not necessarily the only order in which these steps can be performed, and some of the steps described may be omitted and / or some other steps not described herein may be added to the method.

[0077] In a distributed system architecture, when an application malfunctions (such as excessively long transaction processing time or an increased transaction failure rate), traditional troubleshooting methods often struggle to quickly and accurately pinpoint the root cause of the malfunction, resulting in lengthy troubleshooting times and high costs.

[0078] To address the problems existing in the prior art, this invention provides a root cause localization method based on end-to-end topology and multi-dimensional analysis, such as... Figure 1 As shown, it includes the following steps:

[0079] A cross-application horizontal positioning method is established, which includes transaction time-related anomaly positioning and transaction failure anomaly positioning. The transaction time-related anomaly positioning is completed by stripping downstream time-consuming processes, and the transaction failure anomaly positioning is completed by locating the deepest and most frequently hit failure nodes.

[0080] A single-application vertical positioning method is established, which includes application multidimensional drill-down analysis and application-related resource drill-down analysis. The application multidimensional drill-down analysis distinguishes between normal and abnormal sample data, calculates the differences between them in data distribution, feature values, or statistical characteristics, and uses statistical methods to quantify the differences to locate the root cause dimension of the fault. The application-related resource drill-down analysis sets clear trigger conditions for specific indicators based on rules, filters abnormal indicators of the resource model through trigger conditions, and sorts them based on the strength of the dependency model between abnormal indicators and applications. The abnormal indicators of the resource model are displayed and statistically analyzed based on the sorting to recommend and locate the fault.

[0081] Specifically, the methods for locating S1: transaction time-related anomalies are as follows:

[0082] S11: Obtain sample data and preprocess the sample data, which includes the average transaction processing time (transAvgProcTime).

[0083] The preprocessing method is as follows: construct a full-link transaction topology based on the anomaly location of transaction time-related transactions. For example, first query the UID (deduplicated) of 1,000 abnormal transaction details of the current application, and then obtain the transaction details of all applications involved through the deduplicated UID (User Identification). Construct the link based on the transaction details.

[0084] S12: Query abnormal samples based on the application PASO, the three minutes before and the two minutes after the abnormal time point and / or the abnormal time consumption value in the alarm content. Among them, (1) Application PASO is an abbreviation for query tag. When querying abnormal samples, "Application PASO" is used as a filter condition to ensure that the query scope is limited to the specific application where the alarm occurred. (2) The three minutes before and the two minutes after the abnormal time point are specifically: when an alarm is received, query all transaction details (samples) within the time period from "3 minutes before" to "2 minutes after" of the alarm time point. The purpose is to collect data during the fault period for subsequent statistical analysis. (3) The abnormal time consumption value in the alarm content is the core filter condition for screening "abnormal" samples. The alarm itself will contain a trigger threshold (i.e. "abnormal time consumption value"). In this step, query transactions whose actual processing time exceeds the trigger threshold and regard these transactions as "abnormal samples". If the alarm content does not provide a trigger threshold, the default value is used (the default value is, for example, 500ms).

[0085] S13: Based on the queried anomaly samples, locate the anomaly and construct the full-link transaction topology. Subtract the total time of all downstream calls from the total time of the ingress request to obtain the actual time of the application's own business logic, as follows:

[0086] S131: Calculate the application's average execution time using the following formula:

[0087] Among them, T self To determine the actual time taken for a single transaction based on its own business logic, T total T represents the total processing time of the application (including its own logic time and downstream call time). downstream_i Let be the call time of the i-th downstream application, i = 1, 2, ..., n, where n is the number of downstream calls. This is the sum of the times taken by all downstream calls;

[0088] ;in, To calculate the average time taken for the application, m is the total number of samples (e.g., the previously mentioned "1000 abnormal transaction details"), j is the sample index (j = 1, 2, ..., m), each sample corresponds to one independent transaction, and n j This represents the number of downstream calls involved in the j-th sample (the downstream call paths may differ between samples).

[0089] S132: Calculate the absolute deviation of the application's own time consumption. Calculate the deviation (absolute deviation) between the average time consumption of each application in the current alarm period and the average time consumption in the alarm period one week ago, using the following formula:

[0090] Absolute deviation = ;

[0091] and Calculated based on the application's own average time consumption formula; specifically, The average time consumed by the application itself during the current alarm period (calculated based on the end-to-end topology). (where m is the current number of samples). The historical baseline average latency for the same alarm period one week ago (calculated using C historical transaction samples from the same period) is as follows: C represents the current number of samples.

[0092] S133: Calculate the downstream application anomaly impact score (introducing a transaction volume proportional weighting model), the formula is as follows:

[0093] Anomaly impact score = absolute deviation × transaction ratio, where transaction ratio = current downstream application transaction volume / total transaction volume across the entire chain;

[0094] S134: Influence score ranking and recommendation positioning through downstream application anomalies.

[0095] S2: The methods for locating transaction failure exceptions are as follows:

[0096] S21: When abnormal alarms occur in the technical success rate indicator, technical failure rate indicator, and business acceptance rate indicator, S22-S24 are triggered.

[0097] S22: Construct a full-link transaction topology based on failure-type anomaly localization to complete data preprocessing;

[0098] Specifically, firstly, for the abnormal samples (such as 1000 transaction details) found in S12 or S23, obtain their unique transaction IDs (UIDs) and remove duplicates.

[0099] Then, using these deduplicated UIDs, we reverse-engineer all transaction details for all applications involved in these transactions.

[0100] Finally, these detailed data are pieced together to construct a complete call chain (i.e., a full-chain transaction topology). This data processing flow of "building the full chain based on UID" is completely consistent for both time-consuming exceptions and failure exceptions.

[0101] S23: Based on anomalies in technical success rate and / or technical failure rate metrics, combined with the application PASO, the three minutes before and two minutes after the anomaly time point, and the transaction technical status being "failed," query abnormal samples. "Application PASO": As described in S12, this is a filtering condition, limiting it to the application mentioned in the alarm. "Three minutes before and two minutes after": As described in S12, this is a time window, limited to the period when the fault occurred. "Transaction technical status is "failed": This is the core filtering condition, used to find only those transaction records marked as "failed" (e.g., TTS=0) from all transactions within this time window. Summary: The entire query means "query those transactions processed by the alarm application (PASO) within 5 minutes before and after the fault time point, and whose final status is 'failed.'" And / or, based on anomalies in business acceptance rate, combined with the application PASO, the three minutes before and two minutes after the anomaly time point, the transaction technical status being "failed," and the transaction business status being "failed," query abnormal samples. Here, "application PASO" is an abbreviation for the query label.

[0102] S24: Locate the anomaly based on the retrieved samples, as follows:

[0103] S241: Filter out all nodes in the abnormal samples, extract the deepest node on all call paths, and use the deepest failure node as the candidate node for the cause of failure.

[0104] S242: Count the number of times each node records the reason for failure in the transaction data, and select the candidate node with the most failures in the leaf nodes as the root cause node of the anomaly.

[0105] S3: The method of applying multidimensional drill-down analysis is as follows:

[0106] S31: Define the scope of the analysis data as the application transaction detail log;

[0107] S32: Define the trigger condition as all transaction alerts;

[0108] S33: Distinguish between normal sample data and abnormal sample data, calculate the differences between the two in terms of data distribution, feature values ​​or statistical characteristics, and use statistical methods to quantify the differences to locate the root cause dimension of the fault.

[0109] (1) Abnormal sample query conditions:

[0110] Transaction timeout is abnormal: Alarm label (application PASO, etc.) & abnormal time point: three minutes before and two minutes after & greater than or equal to the abnormal timeout value in the alarm content.

[0111] Abnormal transaction volume: Alarm label (application PASO, etc.) & abnormal time point three minutes before and two minutes after & time greater than or equal to 500ms.

[0112] Technical success rate: Alarm labels (applications such as PASO) & abnormal time points: three minutes before and two minutes after the time point & technical failure.

[0113] Business acceptance rate: alarm tags (applications such as PASO) & abnormal time points in the first three minutes and the last two minutes & business failures.

[0114] Technical failure count: Alarm tags (applications such as PASO) & abnormal time points in the three minutes before and two minutes after & technical failures.

[0115] (2) Normal sample query conditions:

[0116] Transaction timeout is abnormal: Alarm label (application PASO, etc.) & abnormal time point is three minutes before and two minutes after & less than the abnormal timeout value in the alarm content.

[0117] Abnormal transaction volume: Alarm label (application PASO, etc.) & abnormal time point three minutes before and two minutes after & time less than 500ms.

[0118] Technical success rate: Alarm label (application PASO, etc.) & three minutes before and two minutes after the abnormal time point & technical success.

[0119] Business acceptance rate: alarm tags (applications such as PASO) & abnormal time points in the three minutes before and two minutes after & business success.

[0120] Technical failures: Alarm tags (applications such as PASO) & abnormal times in the three minutes before and two minutes after & technical successes.

[0121] Note: Alarm labels refer to key elements in alarm work orders, such as the application system identifier PASO.

[0122] Furthermore, statistical methods are used to quantify differences and pinpoint the root causes of failures, as follows:

[0123] S331: Configure transaction statistics dimensions for deviation, including microservice identifier, transaction type, logical data center identifier, IP address, cluster region identifier, region identifier, combined dimensions including transaction type, technical error code and technical error information, and combined dimensions including transaction type, business error code and business error information;

[0124] Calculate the characteristics of each dimension value, including the abnormal traffic percentage and the normal traffic percentage of the current dimension value;

[0125] The formula for calculating the percentage of abnormal traffic for the current dimension value is:

[0126] P （维度值i） This represents the percentage of abnormal traffic for the current dimension value.

[0127] The formula for calculating the normal traffic percentage of the current dimension value is:

[0128] Q （维度值i） This represents the normal traffic percentage for the current dimension value.

[0129] The JS divergence for each dimension is calculated using the following formula:

[0130] Where P and Q represent the probability distributions of abnormal and normal samples in any dimension, respectively. M represents the average distribution of P and Q, KL(P||M) is the KL divergence between P and M, and KL(Q||M) is the KL divergence between Q and M.

[0131] Filtering is performed when the JS divergence is less than 0.1%, and the results are sorted by JS divergence.

[0132] The top 3 values ​​based on the JS divergence, from largest to smallest, are used as the root causes for multidimensional drill-down analysis.

[0133] The core of the "multidimensional drill-down analysis method" is comparative analysis. This invention simultaneously queries both "abnormal samples" (e.g., processing time ≥ 500ms) and "normal samples" (e.g., processing time < 500ms) to calculate the differences in their data distribution. The purpose is to identify which dimension's feature values ​​(e.g., a specific IP address, a specific transaction type TTID) have a significantly higher proportion in "abnormal samples" than in "normal samples." The higher this "deviation," the stronger the correlation between that dimension's feature value and the fault, thus pinpointing the root cause dimension.

[0134] S4: The method of applying the associated resource drill-down analysis is as follows:

[0135] S41: Set explicit trigger conditions for specific indicators based on rules. Specific indicators include status indicators, saturation indicators, and non-capacity indicators. The trigger conditions are as follows:

[0136] Status indicators:

[0137] Simultaneity: Status indicators show a change in status due to anomalies within ten minutes before and after an alarm.

[0138] Anomaly: An alarm is triggered when a status indicator exceeds a specific threshold (e.g., 80%) (the value is greater than 80% in any minute within ten minutes after the alarm).

[0139] In one embodiment, the status indicators include status 0 and status 1, which are used to display the database connection status.

[0140] Saturation index:

[0141] Simultaneity: The saturation index must show a sudden increase (i.e., a significant increase in a short period of time) at the time of the alarm, and the saturation index must show a significant year-on-year increase of 30% between the ten minutes before the alarm and the ten minutes after the alarm (a one-to-one comparison of individual instance indices, avg (after the alarm) vs avg (before the alarm)); if the year-on-year increase between the ten minutes before the alarm and the ten minutes after the alarm is less than 30%, the index is considered normal; otherwise, it is considered an abnormal index.

[0142] Anomaly: An alarm is triggered when the saturation index exceeds a specific threshold (e.g., 80%) (the value is greater than 80% in any minute within ten minutes after the alarm).

[0143] In one embodiment, saturation metrics include CPU utilization, memory utilization, or connection pool utilization.

[0144] Non-capacity metrics:

[0145] Simultaneity: It is required that there is a sudden increase in non-capacity indicators at the time of alarm (i.e., a significant increase in a short period of time), and the non-capacity indicators increase significantly by 30% year-on-year between the ten minutes before the alarm and the ten minutes after the alarm time (one-by-one comparison of indicators of individual instances, avg (after alarm) vs avg (before alarm)).

[0146] Anomalies: A sudden increase in detections compared to historical data (a 30% increase compared to the same period last week (the average of the 15 minutes before and after the alarm) indicates an anomaly).

[0147] In one embodiment, non-capacity metrics include absolute values, such as requests per second and queue depth.

[0148] S42: Filter abnormal indicators of the resource model by trigger conditions;

[0149] S43: Sort based on strong and weak dependency models of abnormal indicators and application associations, and combine the sorting to display and statistically analyze abnormal indicators of resource models for recommendation positioning.

[0150] The present invention also provides a root cause localization system based on end-to-end topology and multi-dimensional analysis, and the root cause localization system is constructed using the root cause localization method.

[0151] In summary, compared with the prior art, the present invention has the following advantages:

[0152] (1) This invention aims to provide a root cause localization method and system based on end-to-end topology and multi-dimensional analysis, so as to shorten the fault diagnosis time, reduce operation and maintenance costs, improve the availability and stability of the system, and achieve:

[0153] 1) Precise location of cross-application anomaly propagation paths;

[0154] 2) Anomaly localization based on multi-dimensional features within a single application;

[0155] 3) Collaborative analysis of anomalies in the resource layer and application layer.

[0156] (2) Improve fault location efficiency: Through automated and intelligent analysis algorithms, the entire transaction topology can be quickly constructed, abnormal samples can be accurately screened, and the root cause of the fault can be quickly located based on the deviation of multiple dimensions. Compared with the traditional manual troubleshooting method, the fault troubleshooting time is greatly shortened and the operation and maintenance efficiency is improved.

[0157] (3) Improve the accuracy of fault location: Multi-dimensional data analysis can more comprehensively and deeply explore the potential information in the data, accurately identify the factors that have a significant impact on the fault, reduce the risk of misjudgment and omission, and more accurately locate the root cause of the fault compared with traditional methods, providing a more reliable basis for fault repair.

[0158] (4) Enhance system stability and availability: timely and accurate fault location and repair reduces the impact time of faults on system operation, improves the overall stability and availability of the system, and ensures business continuity and user satisfaction.

[0159] (5) Reduced operation and maintenance costs: It reduces the workload and time of manual troubleshooting, reduces the workload of operation and maintenance personnel and their dependence on experience, and avoids business losses and potential cost increases caused by long-term unresolved faults, bringing significant cost benefits to the enterprise's IT operation and maintenance.

[0160] The above are merely preferred embodiments of the present invention and do not constitute any limitation on the present invention. Any equivalent substitutions or modifications made by those skilled in the art to the technical solutions and content disclosed in the present invention without departing from the scope of the present invention shall be deemed to have remained within the scope of protection of the present invention.

Claims

1. A root cause positioning method based on full-link topology and multi-dimension analysis, characterized in that, Includes the following steps: A cross-application horizontal positioning method is established, which includes transaction time-related anomaly positioning and transaction failure anomaly positioning. The transaction time-related anomaly positioning is completed by stripping downstream time-consuming processes, and the transaction failure anomaly positioning is completed by locating the deepest and most frequently hit failure nodes. The methods for locating transaction time-related anomalies are as follows: S11: Obtain sample data and preprocess the sample data, which includes the average transaction processing time. S12: Query abnormal samples based on the application PASO, the three minutes before and the two minutes after the abnormal time point and / or the abnormal time consumption value greater than or equal to the alarm content, where application PASO is an abbreviation for query tag; S13: Based on the queried anomaly samples, locate the anomaly and construct the full-link transaction topology. Subtract the total time of all downstream calls from the total time of the ingress request to obtain the actual time of the application's own business logic, as follows: S131: Calculate the application's average execution time using the following formula: Among them, T self To determine the actual time taken for a single transaction based on its own business logic, T total T represents the total processing time of the application. downstream_i Let be the call time of the i-th downstream application, i = 1, 2, ..., n, where n is the number of downstream calls. This is the sum of the times taken by all downstream calls; ;in, The average time taken to apply the method is given, where m is the total number of samples, j is the sample index, and n is the average time taken to apply the method. j Let J be the number of downstream calls involved in the j-th sample. S132: Calculate the absolute deviation of the application's own time consumption, and calculate the deviation between the average time consumption of each application in the current alarm period and the average time consumption in the alarm period one week ago in the sampled logs, as follows: Absolute deviation = ; in, The average time consumed by the application within the current alarm period. The average latency is the historical baseline for the same alarm period one week ago. and Calculated based on the application's own average time consumption formula; S133: Calculate the downstream application anomaly impact score using the following formula: Anomaly impact score = absolute deviation × transaction ratio, where transaction ratio = current downstream application transaction volume / total transaction volume across the entire chain; S134: Influence score ranking and recommendation positioning through downstream application anomalies; The methods for locating transaction failure exceptions are as follows: S21: When abnormal alarms occur in the technical success rate indicator, technical failure rate indicator, and business acceptance rate indicator, S22-S24 are triggered. S22: Construct a full-link transaction topology based on failure-type anomaly localization to complete data preprocessing; S23: Based on anomalies in technical success rate indicators and / or technical failure rate indicators, combined with the application PASO, the three minutes before and the two minutes after the abnormal time point, and the transaction technical status, query abnormal samples for failure; and / or, based on anomalies in business acceptance rate, combined with the application PASO, the three minutes before and the two minutes after the abnormal time point, the transaction technical status is failed, and the transaction business status is failed, query abnormal samples for failure, where application PASO is an abbreviation for query tag; S24: Locate the anomaly based on the retrieved samples, as follows: S241: Filter out all nodes in the abnormal samples, extract the deepest node on all call paths, and use the deepest failure node as the candidate node for the cause of failure. S242: Count the number of times each node records the reason for failure in the transaction data, and take the candidate node with the most failures in the leaf nodes as the root cause node of the anomaly. A single-application vertical positioning method is established, which includes application multidimensional drill-down analysis and application-related resource drill-down analysis. The application multidimensional drill-down analysis distinguishes between normal and abnormal sample data, calculates the differences between them in data distribution, feature values, or statistical characteristics, and uses statistical methods to quantify the differences to locate the root cause dimension of the fault. The application-related resource drill-down analysis sets clear trigger conditions for specific indicators based on rules, filters abnormal indicators of the resource model through trigger conditions, and sorts them based on the strength of the dependency model between abnormal indicators and applications. The abnormal indicators of the resource model are displayed and statistically analyzed based on the sorting to recommend and locate the fault.

2. The root cause localization method based on end-to-end topology and multi-dimensional analysis as described in claim 1, characterized in that, The multidimensional drill-down analysis method is applied as follows: S31: Define the scope of the analysis data as the application transaction detail log; S32: Define the trigger condition as all transaction alerts; S33: Distinguish between normal sample data and abnormal sample data, calculate the differences between the two in terms of data distribution, feature values ​​or statistical characteristics, and use statistical methods to quantify the differences to locate the root cause dimension of the fault.

3. The root cause localization method based on end-to-end topology and multi-dimensional analysis as described in claim 2, characterized in that, The root cause of a failure is identified by quantifying differences using statistical methods, as follows: Configure the transaction statistics dimensions for deviation, including microservice identifier, transaction type, logical data center identifier, IP address, cluster area identifier, region identifier, combined dimensions including transaction type, technical error code and technical error information, and combined dimensions including transaction type, business error code and business error information; Calculate the characteristics of each dimension value, including the percentage of abnormal traffic and the percentage of normal traffic for the current dimension value; The formula for calculating the percentage of abnormal traffic for the current dimension value is: P （维度值i） This represents the percentage of abnormal traffic for the current dimension value. The formula for calculating the normal traffic percentage of the current dimension value is: Q （维度值i） This represents the normal traffic percentage for the current dimension value. The JS divergence for each dimension is calculated using the following formula: Where P and Q represent the probability distributions of abnormal and normal samples in any dimension, respectively. M represents the average distribution of P and Q, KL(P||M) is the KL divergence between P and M, and KL(Q||M) is the KL divergence between Q and M; Filtering is performed when the JS divergence is less than 0.1%, and the results are sorted by JS divergence. The top 3 values ​​based on the JS divergence, from largest to smallest, are used as the root causes for multidimensional drill-down analysis.

4. The root cause localization method based on end-to-end topology and multi-dimensional analysis as described in claim 1, characterized in that, The method of applying the associated resource drill-down analysis is as follows: S41: Set explicit trigger conditions for specific indicators based on rules. Specific indicators include status indicators, saturation indicators, and non-capacity indicators. The trigger conditions are as follows: Status indicators: Simultaneity: Status indicators show a change in status due to anomalies within ten minutes before and after an alarm. Anomalies: An alarm is triggered when a status metric exceeds a specific threshold; Saturation index: Simultaneity: It is required that there is a sudden increase in the saturation index at the time of the alarm, and the saturation index increases significantly by 30% year-on-year between the ten minutes before the alarm and the ten minutes after the alarm time. Anomalies: An alarm is triggered when the saturation index exceeds a specific threshold; Non-capacity metrics: Simultaneity: It requires that there is a sudden increase in non-capacity indicators at the time of alarm, and that the non-capacity indicators increase significantly by 30% year-on-year in the ten minutes before and after the time of alarm. Anomalies: Detecting sudden increases compared to historical data; S42: Filter abnormal indicators of the resource model by trigger conditions; S43: Sort based on strong and weak dependency models of abnormal indicators and application associations, and combine the sorting to display and statistically analyze abnormal indicators of resource models for recommendation positioning.

5. The root cause localization method based on end-to-end topology and multi-dimensional analysis as described in claim 4, characterized in that, Status indicators include status 0 and status 1, which are used to display the database connection status; Saturation metrics include CPU utilization, memory utilization, or connection pool utilization; Non-capacity metrics include absolute values, which include requests per second and queue depth.

6. A root cause localization system based on end-to-end topology and multi-dimensional analysis, characterized in that, A root cause localization system is constructed using the root cause localization method as described in any one of claims 1-5.

Images