An abnormality root cause analysis method and system based on a container management platform

Abstract

This invention discloses a method and system for anomaly root cause analysis based on a container management platform, belonging to the field of data processing technology. The method includes: acquiring an indicator set, which includes multiple indicators; obtaining a correlation coefficient between two indicators based on the Pearson correlation coefficient algorithm; acquiring an abnormal event and a first indicator associated with the abnormal event; obtaining one or more second indicators associated with the first indicator based on the correlation coefficient; determining whether the second indicator is abnormal; if so, the second indicator is the root cause of the anomaly. By using the Pearson correlation coefficient algorithm, the second indicator associated with the abnormal event is calculated, and the root cause / indicator of the anomaly is quickly located within the range of the second indicator, improving the monitoring efficiency of the root cause of the anomaly and the troubleshooting efficiency.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically to an anomaly root cause analysis method and system based on a container management platform. Background Technology

[0002] As the development of business container management platforms (Kubernetes, K8s) accelerates, Kubernetes business metrics data are becoming increasingly standardized. The rapid growth in business volume and data volume brings a large number of abnormal metrics, while scheduling becomes increasingly complex, and anomaly messages become frequent, posing challenges to anomaly localization and root cause analysis.

[0003] Current business system alert monitoring primarily employs a fixed-indicator, full-scale alerting approach. By monitoring all indicators, the business system is inspected and alerts are generated for issues. For example, Prometheus, a standard component of Kubernetes, monitors the current business system, providing statistical analysis of host and container application-level metrics such as memory, CPU, network traffic, disk read / write, and capacity, generating alerts from a resource perspective. However, this monitoring method requires a thorough search of each indicator, resulting in low monitoring efficiency or high computational demands. In scenarios with numerous abnormal business operations, it is difficult to pinpoint the root cause of the anomalies. Summary of the Invention

[0004] To address the aforementioned technical problems in existing technologies, this invention provides an anomaly root cause analysis method and system based on a container management platform. By using the correlation coefficient based on the Pearson correlation coefficient algorithm, highly correlated indicators are obtained, and the highly correlated indicators are analyzed to facilitate rapid location and improve troubleshooting efficiency.

[0005] This invention discloses an anomaly root cause analysis method based on a container management platform. The method includes: acquiring an indicator set, the indicator set including multiple indicators; obtaining a correlation coefficient between two indicators based on the Pearson correlation coefficient algorithm; acquiring an anomaly event and a first indicator associated with the anomaly event; obtaining one or more second indicators associated with the first indicator according to the correlation coefficient; determining whether the second indicator is anomaly; if so, the second indicator is the root cause of the anomaly.

[0006] Preferred method for screening abnormal root causes based on correlation coefficients:

[0007] Get the root cause indicator list;

[0008] Add the aforementioned abnormal root causes to the root cause index list;

[0009] Obtain the absolute value of the correlation coefficient between the abnormal root cause in the root cause index list and the first index;

[0010] Sort the abnormal root causes in descending order of absolute value and obtain the top N abnormal root causes as the results of root cause analysis, where N is a natural number.

[0011] Preferably, alarm information is generated based on the results of the root cause analysis; the alarm information is compressed and then added to the message queue.

[0012] Preferably, the method for establishing the set of correlation coefficients includes:

[0013] Based on the container management platform, obtain the indicator association table and the topology association data table;

[0014] Based on the indicator association table and the topology association data table, obtain the calling and deployment relationships between resources;

[0015] Based on the call relationship and deployment relationship, obtain the set of resource relationships;

[0016] Obtain the resource relationships in the resource relationship set, wherein the resource relationships include a first resource and a second resource;

[0017] Obtain a first indicator instance set of a first resource and a second indicator instance set of a second resource, wherein the first indicator instance set includes one or more first indicator instances and the second indicator instance set includes one or more second indicator instances.

[0018] Obtain historical data for the first indicator instance and the second indicator instance respectively;

[0019] The historical data is analyzed based on the Pearson correlation coefficient algorithm to obtain the correlation coefficient between the first indicator instance and the second indicator instance.

[0020] Traverse the resource relationships and resource relationship set to obtain the set of correlation coefficients.

[0021] Preferably, the formula for calculating the correlation coefficient is expressed as follows:

[0022]

[0023] Where r represents the correlation coefficient, n represents the sample size of the indicator variable, and X i Y i Let X be the current value of the two variables. u Y u Let σ be the mean of the two indicator variables. X σ Y These are the standard deviations of the indicator variables.

[0024] Preferably, the correlation between the indicators is divided based on the absolute value of the correlation coefficient:

[0025] 0.8 < |r| ≤ 1: extremely strong correlation; 0.6 < |r| ≤ 0.8: strong correlation; 0.4 < |r| ≤ 0.6: moderate correlation; 0.2 < |r| ≤ 0.4: weak correlation; 0 ≤ |r| ≤ 0.2: extremely weak correlation or no correlation.

[0026] |r| represents the absolute value of the correlation coefficient.

[0027] Preferred methods for performing root cause analysis on anomalous events include:

[0028] Step 311: Using the set of correlation coefficients, obtain a first indicator associated with the abnormal event;

[0029] Step 312: Determine whether an event has occurred within the valid time frame of the first indicator for the abnormal event;

[0030] If no event occurs, proceed to step 313: Analyze the historical data of the first indicator; if the first indicator is abnormal, proceed to step 314.

[0031] If an event occurs, proceed to step 314: Based on the set of correlation coefficients, obtain one or more second indicators associated with the first indicator;

[0032] Step 315: Determine if the second indicator is abnormal;

[0033] If an anomaly occurs, proceed to step 316: mark the second indicator as the root cause and add it to the root cause indicator list, then proceed to step 317.

[0034] If there are no abnormalities, proceed to step 317: obtain the next second indicator, proceed to step 315, until all second indicators have been judged and the root cause indicator list is obtained.

[0035] Step 318: Select root cause indicators from the root cause indicator list based on the magnitude of the absolute value of the correlation coefficient.

[0036] Preferably, methods for determining whether the second indicator is abnormal include:

[0037] Collect data for the second indicator and record the collection time;

[0038] Obtain the abnormal triggering rules for the second indicator;

[0039] Based on the aforementioned anomaly triggering rules, the data and collection time are analyzed to determine whether the second indicator is abnormal.

[0040] Preferred root cause propagation analysis method:

[0041] Using the set of correlation coefficients, one or more third indicators associated with the second indicator are obtained:

[0042] Determine whether the third indicator meets any of the following conditions: an event occurs within the valid range of the abnormal event, the indicator experiences a data collection interruption, the indicator has abnormal historical data within the valid range of the abnormal event, and it meets the abnormal triggering rules.

[0043] If the conditions are met, the third indicator is considered an abnormal root cause, and it is marked as a root cause indicator. It should be noted that the criteria for determining this abnormal root cause also apply to the first and second indicators.

[0044] The present invention also provides a system for implementing the above-described anomaly root cause analysis method, the system comprising an index relationship analysis module and an anomaly root cause analysis module.

[0045] The indicator relationship analysis module is used to obtain an indicator set and, based on the Pearson correlation coefficient algorithm, obtain the correlation coefficient between two indicators.

[0046] The abnormal root cause analysis module is used to obtain one or more second indicators associated with the first indicator of the abnormal event based on the correlation coefficient, and to determine whether the second indicator is an abnormal root cause based on the abnormality of the second indicator.

[0047] Compared with the prior art, the beneficial effects of the present invention are as follows: by using the Pearson correlation coefficient algorithm, a second indicator associated with the abnormal event is calculated, and the root cause / indicator of the abnormality is quickly located within the range of the second indicator, thereby improving the monitoring efficiency of the root cause of the abnormality and improving the troubleshooting efficiency. Attached Figure Description

[0048] Figure 1 This is a flowchart of the anomaly root cause analysis method based on a container management platform according to the present invention;

[0049] Figure 2 This is a flowchart of the method for performing root cause analysis on abnormal events in Example 2;

[0050] Figure 3 This is the system logic block diagram of the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] The present invention will now be described in further detail with reference to the accompanying drawings:

[0053] An anomaly root cause analysis method based on a container management platform, such as Figure 1 As shown, the method includes:

[0054] Step 101: Obtain the metric set, which includes multiple metrics. Metrics can be obtained from the container management platform.

[0055] Step 102: Obtain the correlation coefficient between the two indicators based on the Pearson correlation coefficient algorithm. The correlation coefficient reflects the correlation between the two indicators.

[0056] Step 103: Obtain the abnormal event and the first indicator associated with the abnormal event.

[0057] Anomalies can be recorded in the container management platform, observed by administrators, or reported by users. They are typically monitored using one or more primary metrics; that is, anomalies are usually accompanied by anomalies in these primary metrics. For example, server outages are often associated with anomalies in CPU utilization and load. Anomalies in these primary metrics are usually quite obvious.

[0058] Step 104: Based on the correlation coefficient, obtain one or more second indicators associated with the first indicator.

[0059] Step 105: Determine whether the second indicator is abnormal.

[0060] Abnormal metrics include slow application access, automatic application restarts in the container, application malfunctions, and related service failures.

[0061] If so, proceed to step 106: The second indicator is the root cause of the anomaly.

[0062] If not, the second indicator is not the root cause of the anomaly.

[0063] Step 107: Iterate through the second index.

[0064] By using the Pearson correlation coefficient algorithm, a second indicator associated with the abnormal event is calculated, and the root cause / indicator of the abnormality is quickly located within the range of the second indicator, thereby improving the monitoring efficiency of the root cause of the abnormality and improving the troubleshooting efficiency.

[0065] For example, in cases of abnormal events such as a decline in the performance of a business application, or application access timeouts, there are usually some abnormal primary indicators, such as abnormal memory indicators, CPU resource fluctuations, or abnormal application threads. In such cases, the root cause of the abnormality can be analyzed and located based on the correlation coefficient.

[0066] The formula for calculating the correlation coefficient is as follows:

[0067]

[0068] Where r represents the correlation coefficient, n represents the sample size of the indicator variable, and X i Y i Let X be the current value of the two variables. u Y u Let σ be the mean of the two indicator variables. X σ Y These are the standard deviations of the indicator variables.

[0069] The correlation coefficient is the quotient of the covariance and standard deviation between two index variables.

[0070] Covariance is expressed as:

[0071]

[0072] Standard deviation is expressed as:

[0073]

[0074] The correlation coefficient is a value between -1 and 1. When the linear relationship between two variables strengthens, the correlation coefficient tends to 1 or -1. When one indicator variable increases, the other indicator variable also increases, indicating that they are positively correlated, and the correlation coefficient is greater than 0. If one indicator variable increases, the other indicator variable decreases, indicating that they are negatively correlated, and the correlation coefficient is less than 0. If the correlation coefficient is equal to 0, it indicates that there is no linear relationship between them.

[0075] In one specific embodiment, the correlation between indicators is classified according to the absolute value of the correlation coefficient: 0.8 < |r| ≤ 1 is extremely strong; 0.6 < |r| ≤ 0.8 is strongly correlated; 0.4 < |r| ≤ 0.6 is moderately correlated; 0.2 < |r| ≤ 0.4 is weakly correlated; and 0 ≤ |r| ≤ 0.2 is extremely weakly correlated or has no correlation, where |r| represents the absolute value of the correlation coefficient.

[0076] You can also filter for abnormal root causes based on the correlation coefficient:

[0077] Step 201: Obtain the root cause indicator list.

[0078] Step 202: Add the abnormal root cause to the root cause index list.

[0079] Step 203: Obtain the absolute value of the correlation coefficient between the abnormal root causes in the root cause index list and the first index.

[0080] Step 204: Sort the anomalies in descending order of absolute value and obtain the top N anomalies as the results of the root cause analysis, where N is a natural number. The absolute value reflects the confidence level of the anomaly root cause. For example, the top 10 anomalies with the largest absolute values ​​are taken as the analysis results. In a specific embodiment, the highest confidence level is set to 100%.

[0081] Step 205: Based on the results of the root cause analysis, generate alarm information; compress the alarm information and add it to a message queue, such as Kafka or MQ. Send and display the results or alarms; further, an anomaly root cause correlation analysis graph can be generated.

[0082] Example 1

[0083] The methods for establishing the set of correlation coefficients include:

[0084] Step 301: Based on the container management platform, obtain the indicator association table and the topology association data table.

[0085] Step 302: Based on the indicator association table and the topology association data table, obtain the calling relationship and deployment relationship between resources.

[0086] Step 303: Obtain the resource relationship set based on the call relationship and deployment relationship.

[0087] Step 304: Obtain the resource relationship between the first resource rel_a and the second resource rel_b in the resource relationship set. The first resource and the second resource constitute the resource relationship.

[0088] Step 305: Obtain the first indicator instance set col_a of the first resource rel_a and the second indicator instance set col_b of the second resource rel_b, wherein the first indicator instance set col_a includes one or more first indicator instances a, and the second indicator instance set col_b includes one or more second indicator instances b.

[0089] Step 306: Obtain the historical data of the first indicator instance a and the second indicator instance b respectively.

[0090] Step 307: Analyze the historical data based on the Pearson correlation coefficient algorithm to obtain the correlation coefficient between the first indicator instance a and the second indicator instance b.

[0091] Step 308: Traverse the resource relationships and resource relationship set to obtain the set of correlation coefficients.

[0092] By finding the first indicator within the set of correlation coefficients, you can quickly locate related second indicators. For example, you can use indicators with a correlation coefficient greater than 0.6 as the second indicator. Then, you can recursively associate the second indicator with the analyzed set of correlation coefficients and analyze the health status of the second indicator.

[0093] Example 2

[0094] like Figure 2 Methods for performing root cause analysis on abnormal events include:

[0095] Step 311: Using the set of correlation coefficients, obtain a first indicator associated with the abnormal event. The first indicator can also be obtained based on data records, experience, user feedback, etc.

[0096] Step 312: Determine whether an event has occurred within the valid range of the first indicator for abnormal events.

[0097] If no event occurs, proceed to step 313: Analyze the historical data of the first indicator. If the first indicator is abnormal, proceed to step 314; if the first indicator is not abnormal, the first indicator is not the root cause of the abnormality.

[0098] If an event occurs, proceed to step 314: Based on the set of correlation coefficients, obtain one or more second indicators associated with the first indicator, and add the first indicator to the root cause indicator list.

[0099] Step 315: Determine if the second indicator is abnormal.

[0100] The methods for determining whether the second indicator is abnormal include:

[0101] Step 501: Collect data for the second metric and record the collection time. Metric data can be obtained through Zabbix, hardware resources, software topology, etc., and sent to Kafka with multiple dimensions and metrics, using the bsm-std-item topic data format.

[0102] The data is specified in the following format: {'level':1, 'triggerId':75, 'env':'dev', 'resId':'25a0', 'platform':'zabbix', 'unitScale':1.0, 'itemId':871, 'recordTime':'2020-05-09 10:33:40', 'metric':'system.cpu.util[user]', 'host':'10.10.103.97', 'name':'CPU user-mode utilization', 'step':60, 'value':'4.3736'}, but not limited to this. When data for a specific metric is received, the latest collection time for that metric is written to Redis in the format itemId:time. This time can be used to determine whether a metric has experienced a data collection interruption.

[0103] Step 502: Obtain the exception triggering rule for the second metric. Since the received message only contains the triggering rule ID ('triggerId' segment) used by the metric, the specific triggering rule needs to be queried based on the ID. To improve query performance, first query Redis; if not found, then query MySQL and write the query result to Redis for future queries.

[0104] Step 503: Based on the aforementioned anomaly triggering rules, analyze the data and collection time to determine whether the second indicator is abnormal. This method is also applicable to determining anomalies in the first indicator. The anomaly triggering rules can be formulated according to actual circumstances.

[0105] If an anomaly occurs, proceed to step 316: mark the second indicator as the root cause and add the root cause to the root cause indicator list, then proceed to step 317.

[0106] If there is no abnormality, the second indicator is not the root cause of the abnormality. Proceed to step 317: obtain the next second indicator, proceed to step 315, until all second indicators have been judged / analyzed and a list of root cause indicators is obtained.

[0107] Step 318: Select root cause indicators from the root cause indicator list based on the magnitude of the absolute value of the correlation coefficient.

[0108] Example 3

[0109] Unlike Example 2, a root cause extension analysis was performed on the third indicator associated with the second indicator:

[0110] Step 401: Using the set of correlation coefficients, obtain one or more third indicators associated with the second indicator.

[0111] Step 402: Determine whether the third indicator meets any of the following conditions: an event occurs within the valid time range of the abnormal event (e.g., within 5-10 minutes when the abnormal event occurs), the indicator has abnormal historical data within the valid range of the abnormal event, and it meets the abnormal triggering rules.

[0112] If satisfied, proceed to step 403, where the third indicator is identified as the root cause of the anomaly, and the third indicator is marked as a root cause indicator.

[0113] If the condition is not met, the third indicator is not a root cause of the anomaly.

[0114] The third indicator is traversed to obtain a list of root causes, and the list of root causes is filtered according to the magnitude of the absolute value of the correlation coefficient.

[0115] However, it is not limited to this. Depending on the actual requirements, indicators at the next or multiple levels can also be monitored, but the amount of calculation is large.

[0116] Example 4

[0117] This embodiment provides a system for implementing the above-described root cause analysis method for anomalies, such as... Figure 3 It includes a monitoring module 1, an abnormal indicator detection module 2, an indicator relationship analysis module 3, an abnormal root cause analysis module 4, and an alarm module 5.

[0118] Monitoring module 1 is used to collect monitoring data; abnormal indicator detection module 2 is used to determine whether the first indicator is abnormal based on the monitoring data.

[0119] The indicator relationship analysis module 3 is used to obtain the indicator set and obtain the correlation coefficient between two indicators based on the Pearson correlation coefficient algorithm.

[0120] The abnormal root cause analysis module 4 is used to obtain one or more second indicators associated with the first indicator of the abnormal event based on the correlation coefficient; and to determine whether the second indicator is an abnormal root cause based on the abnormality of the second indicator.

[0121] Alarm module 5 is used to generate alarms based on the root cause of the anomaly.

[0122] This invention utilizes a root cause analysis method based on a container management platform to break down data silos and achieve comprehensive data processing. Based on heterogeneous data source integration technology, it enables unified processing of massive historical data from various systems, laying a solid foundation for unified monitoring metrics operation. When numerous alarms caused by faults make it difficult to pinpoint the cause, root cause recommendation assists in identifying the root cause from a small number of suspected cases, avoiding inefficient, exhaustive searches and effectively reducing MTTR (mean time to repair), significantly improving the efficiency of the operations and maintenance department. This completely liberates the operations and maintenance department from the tedious, time-consuming, and error-prone analysis of large-scale metric data, allowing them to focus on more valuable business operations.

[0123] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for abnormal root cause analysis based on a container management platform, characterized in that, The method includes: Based on the container management platform, obtain the indicator association table and the topology association data table; Based on the indicator association table and the topology association data table, obtain the calling relationships and deployment relationships between resources; Based on the call relationship and deployment relationship, obtain the set of resource relationships; Obtain the resource relationships between the first and second resources in the resource relationship set; Obtain a first indicator instance set of a first resource and a second indicator instance set of a second resource, wherein the first indicator instance set includes one or more first indicator instances and the second indicator instance set includes one or more second indicator instances. Obtain historical data for the first indicator instance and the second indicator instance respectively; The historical data is analyzed based on the Pearson correlation coefficient algorithm to obtain the correlation coefficient between the first indicator instance and the second indicator instance. Traverse the resource relationships and resource relationship set to obtain the set of correlation coefficients; Obtain an indicator set, which includes multiple indicators; Based on the Pearson correlation coefficient algorithm, the correlation coefficient between the two indicators is obtained; Obtaining an abnormal event and a first indicator associated with the abnormal event includes: using the set of correlation coefficients to obtain a first indicator associated with the abnormal event; Based on the correlation coefficient, one or more second indicators associated with the first indicator are obtained, including: determining whether an event has occurred within the effective time range of the abnormal event for the first indicator; if no event has occurred, analyzing the historical data of the first indicator, and if the first indicator is abnormal, obtaining one or more second indicators associated with the first indicator based on the set of correlation coefficients; if an event has occurred, obtaining one or more second indicators associated with the first indicator based on the set of correlation coefficients. Determine whether the second indicator is abnormal, and if the second indicator is abnormal, mark the second indicator as the root cause and add the root cause to the root cause indicator list; Obtain the next second indicator and determine whether the next second indicator is abnormal, until all second indicators have been judged; Root cause indicators are selected from the list of root cause indicators based on the magnitude of the absolute value of the correlation coefficient.

2. The abnormality root cause analysis method according to claim 1, characterized by, It also includes methods for screening root causes of anomalies based on correlation coefficients: Get the root cause indicator list; Add the aforementioned abnormal root causes to the root cause index list; Obtain the absolute value of the correlation coefficient between the abnormal root cause in the root cause index list and the first index; Sort by absolute value in descending order and retrieve the top [number]. N The abnormal root causes were used as the result of the root cause analysis, among which N It is a natural number.

3. The abnormal root cause analysis method according to claim 2, characterized in that, Based on the results of the root cause analysis, alarm information is generated; the alarm information is compressed and then added to the message queue.

4. The abnormal root cause analysis method according to claim 1, characterized in that, Methods for determining whether the second indicator is abnormal include: Collect data for the second indicator and record the collection time; Obtain the abnormal triggering rules for the second indicator; Based on the aforementioned anomaly triggering rules, the data and collection time are analyzed to determine whether the second indicator is abnormal.

5. The abnormal root cause analysis method according to claim 1, characterized in that, The formula for calculating the correlation coefficient is expressed as: ； in, r Represented as the correlation coefficient, n This represents the sample size as an indicator variable. X i 、Y i These are the current values ​​of the two variables, X u 、Y u These are the means of the two indicator variables, respectively. σ X σ Y These are the standard deviations of the indicator variables.

6. The abnormal root cause analysis method according to claim 5, characterized in that, The correlation between the indicators is classified based on the absolute value of the correlation coefficient: Extremely strong correlation; Strong correlation; Moderately relevant; Weak correlation; Very weak correlation or no correlation It is expressed as the absolute value of the correlation coefficient.

7. The abnormal root cause analysis method according to claim 1, characterized in that, It also includes root cause propagation analysis methods: Using the set of correlation coefficients, one or more third indicators associated with the second indicator are obtained: Determine whether the third indicator meets any of the following conditions or a combination thereof: an event occurs within the valid range of the abnormal event, the third indicator experiences a data loss, the third indicator has abnormal historical data within the valid range of the abnormal event, and it meets the abnormal triggering rules. If the conditions are met, the third indicator is the root cause of the anomaly.

8. A system for implementing the abnormal root cause analysis method as described in any one of claims 1-7, characterized in that, It includes an indicator relationship analysis module and an anomaly root cause analysis module. The indicator relationship analysis module is used to obtain an indicator set and, based on the Pearson correlation coefficient algorithm, obtain the correlation coefficient between two indicators. The abnormal root cause analysis module is used to obtain one or more second indicators associated with the first indicator of the abnormal event based on the correlation coefficient, and to determine whether the second indicator is an abnormal root cause based on the abnormality of the second indicator.

Images