Time series data abnormal root cause analysis method and system

Abstract

The present application discloses a method for analyzing the root causes of abnormalities in time series data, the method comprising: obtaining data to be analyzed from a database according to input time information; preprocessing the data according to preset rules, including data screening and prediction; performing algorithmic analysis on the processed data in multiple search spaces set according to different thresholds to obtain a set of root causes of abnormal points; sorting the set of root causes by frequency, and outputting the root causes with a frequency greater than a preset value as analysis results. The present application also discloses a system for analyzing the root causes of abnormalities in time series data, an electronic device, and a computer-readable storage medium. As a result, the efficiency and accuracy of the root cause analysis of abnormal points in time series data can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of data analysis technology, and in particular to a method, system, electronic device and computer-readable storage medium for analyzing root causes of time series data anomalies. Background Art

[0002] Time series data is a sequence of data points arranged in chronological order. Data points whose values ​​on time series data do not conform to common sense and expectations are called abnormal points. Root cause analysis is often required for abnormal points in time series data. The purpose of root cause analysis is to find the root cause of the problem, which is a method of analyzing and solving problems. It is a structured problem-solving method that is used to gradually find the root cause of the problem and solve it, rather than just focusing on the surface phenomenon of the problem. For example, in business scenarios, sometimes it is necessary to investigate and analyze why the video freeze data is abnormal and where it is abnormal, and seek necessary measures to prevent abnormal accidents from happening again, so as to improve the safety and quality of the overall business.

[0003] Currently, the commonly used root cause analysis algorithms include HotSpot algorithm, Squeeze algorithm, etc. Among them, Squeeze algorithm is an improvement on HotSpot algorithm. However, these existing root cause analysis algorithms are relatively early implementation algorithms, and they use relatively simple methods for time series prediction and clustering algorithms, which cannot guarantee the efficiency and accuracy of root cause analysis of abnormal points. Summary of the invention

[0004] The main purpose of this application is to propose a time series data anomaly root cause analysis method, system, electronic device and computer-readable storage medium, aiming to solve the problem of how to improve analysis efficiency and accuracy.

[0005] To achieve the above purpose, the present invention provides a method for analyzing the root cause of abnormal time series data, the method comprising:

[0006] Acquire the data to be analyzed from the database according to the input time information;

[0007] Preprocessing the data according to preset rules, including data screening and prediction;

[0008] Performing algorithmic analysis on the processed data in multiple search spaces set according to different thresholds to obtain a set of root causes of abnormal points; and

[0009] The root cause set is sorted by frequency, and the root causes whose frequencies are greater than a preset value are output as analysis results.

[0010] Optionally, acquiring the data to be analyzed from the database according to the input time information includes:

[0011] Receive input time information;

[0012] Verify the validity and format correctness of the time information;

[0013] When the verification is passed, the data to be analyzed corresponding to the time information is obtained from the database.

[0014] Optionally, acquiring the data to be analyzed corresponding to the time information from the database includes:

[0015] Set the dimensions of the data to be analyzed based on the business scenario;

[0016] Data corresponding to the time information is obtained from the database according to the dimension.

[0017] Optionally, in a video playback business scenario, the dimensions include provinces, operators, and third-party content distribution network vendors.

[0018] Optionally, the screening process includes:

[0019] Set the key indicators and the weight conditions of the items corresponding to the key indicators;

[0020] The data that does not meet the weight conditions will be eliminated.

[0021] Optionally, the key indicator is the playback freeze rate, the key indicator corresponding items are the number of video freezes and the number of video playbacks, and the weight condition is that the number of video playbacks is greater than a first threshold and the number of video freezes is greater than a second threshold.

[0022] Optionally, the screening process further includes:

[0023] Set time series completeness conditions;

[0024] Data that does not meet the completeness condition will be eliminated.

[0025] Optionally, the completeness condition includes: there is no missing data in the analysis point and the true value is not 0.

[0026] Optionally, the completeness condition also includes: the predicted value of the analysis point is greater than a third threshold and less than a fourth threshold.

[0027] Optionally, the prediction process includes:

[0028] Setting a plurality of time series prediction algorithms and a weight of each of the time series prediction algorithms;

[0029] Get the true value of time series data in each dimension;

[0030] Using the multiple time series prediction algorithms to predict the true values ​​respectively to generate corresponding predicted values;

[0031] The predicted value and the weight obtained by each of the time series prediction algorithms are combined to calculate the final predicted value.

[0032] Optionally, performing algorithmic analysis on the processed data in a plurality of search spaces set according to different thresholds to obtain a set of root causes of abnormal points includes:

[0033] Converting the processed data into a data format required by a root cause analysis algorithm;

[0034] Setting multiple possible values ​​of multiple thresholds required by the root cause analysis algorithm;

[0035] Setting a plurality of search spaces according to the plurality of thresholds and the plurality of possible values;

[0036] Performing a root cause analysis on the processed data using the root cause analysis algorithm in each search space to obtain a root cause analysis result;

[0037] The root cause set is obtained by integrating the root cause analysis results of the multiple search spaces.

[0038] Optionally, the method further comprises:

[0039] When performing root cause analysis on the processed data, the number of clustering intervals is adjusted according to the data volume of the business scenario.

[0040] Optionally, the root cause analysis algorithm is a Squeeze algorithm.

[0041] In addition, to achieve the above purpose, the embodiment of the present application also provides a time series data abnormal root cause analysis system, the system comprising:

[0042] An acquisition module is used to acquire the data to be analyzed from the database according to the input time information;

[0043] A processing module, used for preprocessing the data according to preset rules, including data screening and prediction;

[0044] An analysis module is used to perform algorithmic analysis on the processed data in multiple search spaces set according to different thresholds to obtain a set of root causes of abnormal points;

[0045] The output module is used to sort the root cause set by frequency and output the root cause whose frequency is greater than a preset value as the analysis result.

[0046] To achieve the above-mentioned purpose, an embodiment of the present application also provides an electronic device, which includes: a memory, a processor, and a time series data anomaly root cause analysis program stored in the memory and executable on the processor, wherein the time series data anomaly root cause analysis program implements the time series data anomaly root cause analysis method as described above when executed by the processor.

[0047] To achieve the above objectives, an embodiment of the present application also provides a computer-readable storage medium, on which a time series data anomaly root cause analysis program is stored. When the time series data anomaly root cause analysis program is executed by a processor, the time series data anomaly root cause analysis method as described above is implemented.

[0048] The time series data anomaly root cause analysis method, system, electronic device and computer-readable storage medium proposed in the embodiments of the present application provide a solution for automatically analyzing the possible causes of abnormal points in time series data, which can reduce a lot of manual analysis workload (saving time cost) and can also more efficiently and continuously output a standardized result (automatically output standardized data), facilitate query statistics and analysis (storage and subsequent acquisition of data), and effectively improve the efficiency and accuracy of root cause analysis of abnormal points in time series data. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 An application environment architecture diagram for implementing various embodiments of the present application;

[0050] Figure 2 A flowchart of a method for analyzing root causes of abnormalities in time series data proposed in the first embodiment of the present application;

[0051] Figure 3 for Figure 2 Detailed flow chart of step S20;

[0052] Figure 4 for Figure 3 Detailed flow chart of step S204;

[0053] Figure 5 for Figure 2 A first detailed flow chart of step S22;

[0054] Figure 6 for Figure 2 A second detailed flow diagram of step S22;

[0055] Figure 7 for Figure 2 Detailed flow chart of step S24;

[0056] Figure 8A flowchart of a specific embodiment of the method for analyzing the root cause of abnormality of time series data is implemented through the cooperation between multiple scripts;

[0057] Fig. 9 A schematic diagram of the hardware architecture of an electronic device proposed in the second embodiment of the present application;

[0058] Fig.10 This is a module diagram of a time series data anomaly root cause analysis system proposed in the third embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0060] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0061] See also Figure 1 , Figure 1 The present invention is applicable to an application environment including, but not limited to, a client 2 , a server 4 , and a database 6 .

[0062] The client 2 is used to display the interface of the current application to the user and receive the user's operation, such as receiving the date to be queried input by the user and transmitting it to the server 4 (of course, it can also be set as a scheduled task, for example, the server 4 regularly queries the analysis results of the previous day every day, and the user does not need to enter the date), receiving the root cause analysis results of the time series data anomalies for the date returned by the server 4 and displaying the results to the user, etc. The client 2 can be a terminal device such as a PC (Personal Computer), a mobile phone, a tablet computer, a portable computer, etc.

[0063] The server 4 is used to provide data and technical support for the client 2, for example, to obtain corresponding data from the database 6 according to the date, perform abnormal root cause analysis on the data, and return the analysis result to the client 2. The server 4 can be a computing device such as a rack server, a blade server, a tower server or a cabinet server, and can be an independent server or a server cluster composed of multiple servers.

[0064] The database 6 is used to store data of various business scenarios. Taking the abnormal root cause analysis of the video playback freeze rate as an example, the database 6 can store data such as the (playing) province, operator, third-party CDN (Content Delivery Network) manufacturer, number of plays, number of freezes, and play time corresponding to the video.

[0065] The client 2, server 4 and database 6 are connected to each other via a wired or wireless network for data transmission and interaction. It is worth noting that the database 6 can be located in the server 4 or exist separately in other electronic devices.

[0066] Embodiment 1

[0067] like Figure 2 As shown, it is a flowchart of a method for analyzing the root cause of abnormal time series data proposed in the first embodiment of the present application. It can be understood that the flowchart in the embodiment of the present method is not used to limit the order of executing the steps. As needed, some steps in the flowchart can also be added or deleted. The method is described below with the server 4 as the execution subject.

[0068] The method comprises the following steps:

[0069] S20, acquiring the data to be analyzed from the database according to the input time information.

[0070] For more details, see Figure 3 , is a detailed flow chart of the above step S20. In this embodiment, the step S20 specifically includes:

[0071] S200, receiving input time information.

[0072] The time information may be the date to be queried, which may be input by the user in the client or automatically input by a scheduled task. For example, when the root cause analysis result of the abnormal value of the jam rate on June 29, 2021 is to be queried, the input date "June 29, 2021" is received.

[0073] S202, verifying the validity and format correctness of the time information.

[0074] Before analyzing the data corresponding to the time information, it is first necessary to verify the validity and format correctness of the time information.

[0075] S204, when the verification is passed, obtaining the data to be analyzed corresponding to the time information from the database.

[0076] When the verification result is that the time information is valid and in the correct format, the verification is passed, and the data to be analyzed corresponding to the time information is obtained from the database. In this embodiment, in order to reduce the amount of data and ensure the credibility of the analysis results, it is necessary to actively reduce the dimension when obtaining the data.

[0077] For more details, see Figure 4 , is a detailed flow chart of the above step S204. In this embodiment, the step S204 specifically includes:

[0078] S2040, set the dimensions of the data to be analyzed according to the business scenario.

[0079] The setting of dimensions is the selection and processing of data sources. This embodiment can perform root cause analysis for business scenarios such as video freezes, first frame anomalies, and loading failures. Prior to this, different representative dimensions can be investigated, screened, and generated for the business scenarios in advance. Since the amount of original data is too large (possibly over 100 million), it will bring serious performance problems. This embodiment actively discards dimensions that cannot be quantified, repeated similar dimensions, overly complex dimensions, and dimensions with too many elements, giving priority to performance while leaving dimensions with sufficient explanatory power. The combination of dimensions with sufficient explanatory power and the fluctuation curve of data anomalies have large overlapping events and similar fluctuation degrees, and can locate the root cause of data anomalies. Explainability explains the degree of understanding of model decisions or analysis results, so sufficient explainability is needed to improve the comprehensibility of the results, thereby improving the trustworthiness of the analysis results.

[0080] For example, for the business scenario of analyzing the root cause of video jamming, the dimension can be set to ['province & operator & third-party CDN manufacturer']. In the subsequent analysis results, dimension combinations such as ['Shandong & China Mobile & Huawei'] may appear, indicating that the abnormal jamming rate at this point may be caused by an abnormality in the Huawei CDN machine node of the Shandong Mobile operator.

[0081] S2042: Acquire data corresponding to the time information from the database according to the dimension.

[0082] After setting the dimensions of the data to be analyzed, the corresponding data is obtained from the database according to the dimensions and the time information for root cause analysis. Specifically, the data corresponding to the time information is first obtained from the database, and then the corresponding data is aggregated based on the dimensions and time (such as date), and the final data is obtained. For example, each piece of data in the database may include more than a dozen dimensions, but only a few dimensions remain after aggregation, removing unused dimensions, avoiding long program running time caused by excessive data volume.

[0083] Back to Figure 2 , S22, pre-processing the data according to preset rules.

[0084] In this embodiment, the preprocessing includes but is not limited to two aspects: data screening and prediction.

[0085] The data screening mainly refers to screening the data obtained from the database according to the value KPI (Key Performance Indicator, which refers to the key indicator of root cause analysis in this embodiment and is a numerical value) weight and time series integrity to eliminate interference data. The prediction mainly refers to obtaining the corresponding prediction value using a prediction algorithm based on the true value of the time series data.

[0086] For more details, see Figure 5 , is a first detailed flow chart of the above step S22. In this embodiment, the step S22 specifically includes:

[0087] S220, setting key indicators and weight conditions of items corresponding to the key indicators.

[0088] In this embodiment, the quantitative KPI weight is considered to reduce the algorithm time consumption and improve accuracy. The quantitative KPI (key indicator) is a dimension that describes the business scenario as the KPI value. For example, in the root cause analysis of video playback jamming, the dimension of playback jamming rate is selected as the quantitative KPI.

[0089] At the same time, the items related to the key indicators are used as weight items, and dynamic thresholds are set to filter out KPIs with values ​​that are too small, thereby eliminating weak weight data and leaving candidate root cause combinations with higher influence and explanatory power, greatly reducing the time consumption of the analysis process and increasing the probability of the real root cause being analyzed. For example, the numerical values ​​of the numerator and denominator of the playback jam rate (that is, the number of video jams and the number of video playbacks) are used as weight items, and the weight conditions are set to simultaneously meet the number of playbacks greater than 1,000 times (first threshold) and the number of jams greater than 10 (second threshold). Data that meets the weight conditions is higher weight data, and low weight data is eliminated.

[0090] S221, eliminating data that does not meet the weight condition.

[0091] If some of the data obtained from the database do not meet the weight conditions, these data will be eliminated and only the data meeting the weight conditions will be retained.

[0092] S222, setting time series completeness conditions.

[0093] In this embodiment, the time series integrity is also considered to further improve the analysis efficiency and the accuracy of the analysis results. Before entering the root cause analysis algorithm for analysis, in order to exclude interference items and error root causes, a condition set is added to judge and process the time series integrity.

[0094] Specifically, the completeness condition includes that there is no missing data in the analysis time window (a moving window of time series data, where the window length is a preset parameter) / analysis point, and the true value of the data is not 0. If there is missing data in the analysis time window, or the true value of the analysis point is 0, the time series data is judged to be incomplete. The analysis point refers to the data in the analysis time window.

[0095] In a preferred embodiment, the completeness condition also includes that the predicted value of the analysis point is greater than a third threshold (for example, 1 / 10000) and less than a fourth threshold (the following threshold thre). When the predicted value of the analysis point is too small (for example, less than 1 / 10000), or the predicted value far exceeds the fourth threshold, it is also judged as an incomplete time series (judged and screened after the predicted value is obtained through the prediction algorithm later). If the predicted value of the analysis point is too small, it means that the predicted values ​​of all data in the analysis time period window are too small, or the predicted values ​​of one or more data are extremely small; if the predicted value of the analysis point far exceeds the fourth threshold, it also means that there are data anomalies in the analysis time period window (the actual value of the data itself is abnormal or the prediction process is abnormal, etc.), therefore, both of these situations should be eliminated as abnormal data.

[0096] S223, eliminating data that does not meet the integrity condition.

[0097] For the time series data that does not meet the completeness condition, it means that the time series data is incomplete and needs to be eliminated.

[0098] At this point, two rounds of data screening have been completed based on the quantitative KPI weights and completeness conditions.

[0099] In this embodiment, the root cause analysis is performed on the filtered data based on the Squeeze algorithm. The Squeeze algorithm is an improvement on the HotSpot algorithm. The HotSpot algorithm is a root cause analysis algorithm published by Tsinghua NetMan Lab in IEEE in 2018. It proposes a hypothesis Ripple Effect: if the dimension combination (A=a, *, *) is abnormal, then the dimension combination (A=a, B=b, C=c) will show abnormal changes in the same proportion. Based on the Ripple Effect, it proposes an indicator Potential Score to evaluate whether a dimension combination set is the root cause. Because the search space is generally very large and cannot be searched completely, the HotSpot algorithm uses MCTS (Monte Carlo Tree Search) for a more efficient search to search for the dimension combination set with the largest Potential Score. The Squeeze algorithm is a root cause analysis algorithm published by Tsinghua NetMan Lab in ISSRE2019. The basic idea is still Ripple Effect and Potential Score, but both have been improved to Generalized Ripple Effect and Generalized Potential Score, which are more generalized and can adapt to more actual situations. Its search is a heuristic method that can get results that are good enough and stable enough in a guaranteed time.

[0100] To use the Squeeze algorithm, the real value and predicted value of the current point of the time series data are required. The real value is obtained and generated from the database in the previous step (S20), and the predicted value needs to be obtained through the time series prediction algorithm. In this embodiment, multiple time series prediction algorithms such as MA (Moving Average), EWMA (Exponentially Weighted Moving Average), and ARIMA (Auto Regressive Integrated Moving Average) are combined to generate predicted values.

[0101] Further reading Figure 6, is a second detailed flow chart of the above step S22. In this embodiment, the step S22 also includes:

[0102] S224, setting a plurality of time series prediction algorithms and a weight of each of the time series prediction algorithms.

[0103] In this embodiment, three time series prediction algorithms, MA, EWMA, and ARIMA, can be used. A weight value is set for each of the three algorithms, and the sum of the three weight values ​​is 1. Since each time series prediction algorithm is predicted, but each focuses on different aspects, in this embodiment, the weight of each algorithm is allocated according to the specific business scenario, and the final prediction value is formed by combining the prediction data obtained by the three algorithms.

[0104] The way to assign weight values ​​can be to judge the fluctuation pattern of the time series, for example, to judge whether the fluctuation of the time series is more affected by periodicity, or more affected by seasonality, or more affected by the past few days, and then calculate a weight value for each of the three algorithms according to the judgment result, and the sum of the weight values ​​is 1.

[0105] S225, obtaining the true value of the time series data of each dimension.

[0106] After filtering the data obtained from the database according to the KPI weight and the time series completeness, the true value of the time series data of each dimension can be obtained according to the filtered data. Of course, the true value of the time series data of each dimension can also be obtained for the data obtained from the database first, and then the data can be filtered after prediction.

[0107] For example, assuming that the data of a video from June 1, 2021 to June 28, 2021 is obtained from the database, and the dimension is ['province & operator & third-party CDN manufacturer']. After completing the screening, for the specific dimension combination of ['Shandong & Mobile & Huawei'], the true value of the time series data of the freeze rate from June 1, 2021 to June 28, 2021 can be obtained.

[0108] S226, respectively using the multiple time series prediction algorithms to make predictions based on the true values ​​to generate corresponding predicted values.

[0109] Among them, the moving window length of the time series is a preset parameter. For example, assuming that the moving window length is five days, based on the true value of the time series data of the jam rate from June 24, 2021 to June 28, 2021, the three time series prediction algorithms of MA, EWMA, and ARIMA are used to predict the jam rate on June 29, 2021, and three prediction values ​​are obtained.

[0110] S227, combining the predicted values ​​and the weights obtained by each of the time series prediction algorithms to calculate a final predicted value.

[0111] Finally, the predicted values ​​obtained by each time series prediction algorithm are multiplied by the corresponding weight value, and then added to obtain the final predicted value. In this way, a time series prediction value that takes into account the influence of the weights of various algorithms can be obtained. The final predicted value and the true value are used as the data required by the root cause analysis algorithm model for format conversion.

[0112] It is worth noting that the data screening and data prediction processes are two aspects of the preprocessing, and the order of execution can be interchanged or performed crosswise, which will not be described in detail here. In fact, when eliminating data that does not meet the conditions according to the completeness condition in the data screening, the data prediction process needs to be performed first.

[0113] Back to Figure 2 , S24, performs algorithmic analysis on the processed data in multiple search spaces set according to different thresholds to obtain a set of root causes of abnormal points.

[0114] This embodiment adds considerations of parameter grid and root cause frequency when performing root cause analysis. A grid search is performed on the parameters used when performing root cause analysis using the Squeeze algorithm (e.g., the threshold thre used to filter out abnormal root causes and the threshold theta used to screen the likelihood score during clustering) to obtain root cause analysis results for different parameter combinations. For example, if the threshold thre is set to 3 possible values ​​and the threshold theta is set to 3 possible values, the grid search will perform 3*3=9 calculations, and the Squeeze algorithm is used once for each calculation.

[0115] For more details, see Figure 7 , is a detailed flow chart of the above step S24. In this embodiment, the step S24 specifically includes:

[0116] S240, converting the processed data into a data format required by a root cause analysis algorithm.

[0117] That is, the final predicted value and the true value are converted into the data format required by the Squeeze algorithm. For example, the processed data may include multiple field names (such as province, operator, third-party CDN manufacturer) and specific content under each field, and according to the requirements of the Squeeze algorithm, these data need to be converted into a data format such as field A (A1, A2, ...), field B (B1, B2, ...).

[0118] S242: Setting multiple possible values ​​of multiple thresholds required by the root cause analysis algorithm.

[0119] In this embodiment, it includes but is not limited to setting multiple possible values ​​of the threshold thre for filtering out abnormal root causes and the threshold theta for screening possibility scores during clustering (for example, 3 possible values ​​are set for each). Among them, the threshold thre is the threshold that the predicted value cannot exceed in the above-mentioned time series integrity condition, and the threshold theta is a threshold that comes with the Squeeze algorithm. The different possible values ​​of each threshold determine the degree of strictness of the judgment condition during root cause analysis (generally, the smaller the possible value of the threshold, the looser the condition; conversely, the greater the possibility of the threshold, the stricter the condition). By setting multiple possible values ​​of the multiple thresholds and then combining them, analysis results under different degrees of judgment conditions can be obtained.

[0120] It is worth noting that sometimes a calculation ratio is set for certain thresholds, and the specific threshold needs to be calculated based on the actual data volume and the calculation ratio.

[0121] S244, setting multiple search spaces according to the multiple thresholds and the multiple possible values.

[0122] By combining multiple possible values ​​of the multiple thresholds, multiple search spaces can be obtained. For example, if the threshold thre is set to 3 possible values ​​and the threshold theta is set to 3 possible values, and the possible values ​​of the threshold thre and the threshold theta are combined, 3*3=9 different search spaces can be set.

[0123] S246, performing a root cause analysis on the processed data using the root cause analysis algorithm in each of the search spaces to obtain a root cause analysis result.

[0124] In the multiple search spaces set in the previous step, each search space uses the Squeeze algorithm to perform a root cause analysis based on the final predicted value and the true value to obtain a root cause analysis result. For example, the Squeeze algorithm is used to perform root cause analysis in 9 search spaces, and a total of 9 root cause analysis results are obtained.

[0125] S248, obtaining the root cause set by integrating the root cause analysis results of the multiple search spaces.

[0126] The root cause analysis results of all the multiple search spaces are combined to obtain the root cause set. For example, the root cause set finally obtained from the above 9 search spaces is a combination of 9 root cause analysis results.

[0127] Back to Figure 2, S26, sorting the root cause set by frequency, and outputting the root cause whose frequency is greater than a preset value as the analysis result.

[0128] For the same batch of data, the frequency of different root cause analysis results obtained by grid search is counted, and then sorted in descending order. The results of multiple analyses with different parameter combinations are used to replace the results of one analysis, and the results corresponding to the frequency that does not reach the preset value are screened out.

[0129] For example, in the root cause set obtained from the above 9 search spaces, the frequency of occurrence of each root cause is counted and then compared with a preset value (for example, 9 times). The root cause whose frequency reaches (greater than or equal to, for each search space) the preset value is returned as the final analysis result, and the root cause whose frequency does not reach the preset value is excluded.

[0130] In addition, this embodiment also optimizes the clustering results for the Squeeze algorithm. The Squeeze algorithm performs clustering based on continuous non-zero intervals and is sensitive to the numerical values ​​of the clustering intervals. The clustering interval means that when performing data clustering, the data of each dimension must first be divided into equal intervals, and these intervals are clustering intervals. Therefore, the coarse and fine granularity of clustering can be determined by adjusting the number of intervals (how many clustering intervals the data is divided into). In this embodiment, the amount of data in the business scenario can be used as an optimization target to adjust the number of clustering intervals.

[0131] In the current prior art, the root cause analysis of abnormal points requires handwriting SQL each time to determine whether abnormalities occur in different dimensions, which is very time-consuming and also involves a lot of repeated processes. This embodiment redevelops and optimizes the Squeeze algorithm from many aspects, such as dimensionality reduction, active dimension selection, hierarchical dimension drilling, clustering algorithm optimization, root cause result tuning, multi-indicator dimensional analysis, etc., so that the method can be applied to the root cause analysis of quality data such as video freeze, first frame, and loading failure.

[0132] The method for analyzing the root cause of anomalies in time series data proposed in this embodiment is optimized based on the Squeeze algorithm, and provides a solution for automatically analyzing the possible causes of abnormal points in time series data. It can reduce a lot of manual analysis workload (saving time cost), and can also more efficiently and continuously output a standardized result (automatically output standardized data), facilitate query statistics and analysis (data storage and subsequent acquisition), and effectively improve the efficiency and accuracy of root cause analysis of abnormal points in time series data.

[0133] In order to explain the above steps of the method in more detail, a specific business scenario is used as an example for explanation. Those skilled in the art should know that the content of the following specific embodiments is not intended to limit the inventive concept of the present invention, and those skilled in the art can easily carry out appropriate content divergence and expansion based on the specific description of the following embodiments.

[0134] Business scenario 1: Jam analysis

[0135] (1) Assume that the jam rate value on June 29, 2021 is abnormal. First, enter the date of June 29, 2021, and obtain the data of the corresponding date from the database. The data includes dimensions such as province, operator, third-party CDN manufacturer, number of jammed videos, number of playbacks, number of jams, playback time, and jam rate. It is aggregated based on date, province, operator, and third-party CDN manufacturer, avoiding long program running time caused by excessive data volume.

[0136] (2) The three dimensions ['province & operator & third-party CDN manufacturer'] were selected as the data dimensions for this analysis, and the freeze rate was used as the KPI indicator. The preconditions such as the platform where the video is located, the country, the video format, and the network used were set. Then, the parameters such as the time series moving window size, the threshold calculation ratio, and the threshold size were set.

[0137] (3) Use the number of playbacks and the number of videos with stuck videos as the weight items of the KPI indicators. Set the data with higher weights when both the number of playbacks is greater than 1,000 and the number of videos with stuck videos is greater than 10, and eliminate the data with low weights.

[0138] (4) Determine the completeness of the time series based on the set completeness conditions and eliminate data that do not meet the completeness conditions.

[0139] (5) The predicted value is a combination of the forecast data obtained by the MA, EWMA, and ARIMA algorithms. The predicted value and the true value are used as the data required by the Squeeze algorithm model and are formatted.

[0140] (6) Set possible value combinations of the threshold thre used to filter out abnormal root causes and the threshold theta used to filter the probability score during clustering (here, each has 6 possible values), and use the same batch of data to calculate all possible values ​​of the combination to obtain a batch of root cause analysis results. For these results, sort them in descending order according to the frequency of occurrence of the dimension combination, calculate a frequency screening preset value (for example, 36 or 30), and display the results that reach the preset value.

[0141] For example, after limiting the video format to on-demand, the platform to Android, the network to WIFI, and the country to China, the analysis results for the sample date are ['Hubei & Telecom & Qiniu', 'Hunan & Telecom & Qiniu', 'Shanghai & Telecom & Qiniu'], which can be used as a preliminary analysis result for further verification and analysis. For example, if Telecom and Qiniu appear in the combination of these three dimensions, you can verify whether it is a problem with the third-party CDN manufacturer Qiniu or a problem with Telecom.

[0142] Business scenario 2: Loading failure analysis

[0143] (1) Assume that the loading failure rate on June 27, 2021 is abnormal. First, enter the date of June 27, 2021, and obtain the data for the corresponding date from the database. The data includes dimensions such as province, operator, third-party CDN manufacturer, number of loading failures, number of plays, and loading failure rate. It is aggregated based on date, province, operator, and third-party CDN manufacturer, avoiding long program running time caused by excessive data volume.

[0144] (2) The three dimensions ['province & operator & third-party CDN manufacturer'] were selected as the data dimensions for this analysis, and the loading failure rate was used as the KPI indicator. The preconditions such as the video platform, country, video format, and network used were set, and then the parameters such as the time series moving window size, threshold calculation ratio, and threshold size were set.

[0145] (3) Use the number of loading failures and the number of video plays as the weight items of the KPI indicators. Set the data with higher weights when both the number of plays is greater than 1,000 and the number of loading failures is greater than 5, and eliminate the data with low weights.

[0146] (4) Determine the completeness of the time series based on the set completeness conditions and eliminate data that do not meet the completeness conditions.

[0147] (5) The predicted value is a combination of the forecast data obtained by the MA, EWMA, and ARIMA algorithms. The predicted value and the true value are used as the data required by the Squeeze algorithm model and are formatted.

[0148] (6) Set possible value combinations of the threshold thre used to filter out abnormal root causes and the threshold theta used to filter the probability score during clustering (here, each has 6 possible values), and use the same batch of data to calculate all possible values ​​of the combination to obtain a batch of root cause analysis results. For these results, sort them in descending order according to the frequency of occurrence of the dimension combination, calculate a frequency screening preset value (for example, 36 or 30), and display the results that reach the preset value.

[0149] For example, after limiting the video format to on-demand, the platform to Android, the network to WIFI, and the country to China, the analysis results for the sample date are ['Heilongjiang & Telecom & mCDN_Mingfu Cloud', 'Chongqing & Mobile & other', 'Anhui & Mobile & other'], which can be used as a preliminary analysis result for further verification and analysis. The element other, which represents other third-party CDN vendors, is generally not considered because it has a small bandwidth and is difficult to analyze specifically, so the possible problems of mCDN_Mingfu Cloud can be mainly considered.

[0150] Business scenario three: first frame anomaly analysis

[0151] (1) Assume that the first frame data on June 30, 2021 is abnormal. First, enter the date of June 30, 2021, and obtain the data of the corresponding date from the database. The data includes dimensions such as province, operator, third-party CDN manufacturer, number of plays, first frame time data, first packet time data, and connection time. It is aggregated based on date, province, operator, and third-party CDN manufacturer, avoiding the long program running time caused by excessive data volume.

[0152] (2) The three dimensions ['province & operator & third-party CDN manufacturer'] were selected as the data dimensions for this analysis, and the first frame time data was used as the KPI indicator. The preconditions such as the video platform, country, video format, and network used were set, and then the parameters such as the time series moving window size, threshold calculation ratio, and threshold size were set.

[0153] (3) Use the number of plays as the weight item of the KPI indicator, set the data with a play count of more than 1,000 times as a higher weight data, and eliminate the data with low weight.

[0154] (4) Determine the completeness of the time series based on the set completeness conditions and eliminate data that do not meet the completeness conditions.

[0155] (5) The predicted value is a combination of the forecast data obtained by the MA, EWMA, and ARIMA algorithms. The predicted value and the true value are used as the data required by the Squeeze algorithm model and are formatted.

[0156] (6) Set possible value combinations of the threshold thre used to filter out abnormal root causes and the threshold theta used to filter the probability score during clustering (here, each has 6 possible values), and use the same batch of data to calculate all possible values ​​of the combination to obtain a batch of root cause analysis results. For these results, sort them in descending order according to the frequency of occurrence of the dimension combination, calculate a frequency screening preset value (for example, 36 or 30), and display the results that reach the preset value.

[0157] For example, after limiting the video format to on-demand, the platform to Android, the network to WIFI, and the country to China, the analysis result of the sample date is [('Hebei & China Unicom', 'Significant first frame abnormality', 'Normal connection & normal first packet')], which can be used as a preliminary analysis result for further verification and analysis. No specific third-party CDN vendors appear here, indicating that the data of many CDN vendors has deteriorated, which may be related to the region and operator. At the same time, the connection time and the first packet time are normal, so the first frame-related issues can be considered.

[0158] See also Figure 8 As shown, it is a flowchart of a specific embodiment of implementing the time series data abnormality root cause analysis method through the cooperation between multiple scripts.

[0159] exist Figure 8 In the system, Shell scripts are mainly used to manage scheduled tasks. The root cause analysis API (Application Programming Interface) is called at a fixed time every day, and the input date is the day before the current date.

[0160] After receiving the date passed in by the Shell script from the API, the Java project is mainly used to verify the existence and validity of the date, call the Python script, pass in the date, and then obtain the root cause analysis result of the Python script, and then return it to the Shell script.

[0161] The Python script is mainly used to obtain the corresponding data from the database according to the date passed in by the Java project, and perform root cause analysis of the abnormal point according to the data (the process of the first embodiment above), and return the analysis results to the Java project. The specific implementation process of each step is described in the first embodiment above, which will not be repeated here.

[0162] Embodiment 2

[0163] like Fig. 9 FIG. 2 is a schematic diagram of the hardware architecture of an electronic device 20 according to the third embodiment of the present application. In this embodiment, the electronic device 20 may include, but is not limited to, a memory 21, a processor 22, and a network interface 23 that can be interconnected via a system bus. It should be noted that Fig. 9 Only the electronic device 20 having components 21 - 23 is shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. In this embodiment, the electronic device 20 may be the server 4 .

[0164] The memory 21 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 21 can be an internal storage unit of the electronic device 20, such as a hard disk or memory of the electronic device 20. In other embodiments, the memory 21 can also be an external storage device of the electronic device 20, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the electronic device 20. Of course, the memory 21 can also include both the internal storage unit of the electronic device 20 and its external storage device. In this embodiment, the memory 21 is generally used to store the operating system and various application software installed in the electronic device 20, such as the program code of the time series data abnormal root cause analysis system 60, etc. In addition, the memory 21 can also be used to temporarily store various types of data that have been output or are to be output.

[0165] The processor 22 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 22 is generally used to control the overall operation of the electronic device 20. In this embodiment, the processor 22 is used to run the program code stored in the memory 21 or process data, such as running the time series data abnormality root cause analysis system 60.

[0166] The network interface 23 may include a wireless network interface or a wired network interface. The network interface 23 is generally used to establish a communication connection between the electronic device 20 and other electronic devices.

[0167] Embodiment 3

[0168] like Fig.10 As shown, a module diagram of a time series data anomaly root cause analysis system 60 is proposed in the third embodiment of the present application. The time series data anomaly root cause analysis system 60 can be divided into one or more program modules, one or more program modules are stored in a storage medium, and are executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment.

[0169] In this embodiment, the time series data abnormality root cause analysis system 60 includes:

[0170] The acquisition module 600 is used to acquire the data to be analyzed from the database according to the input time information.

[0171] The processing module 602 is used to pre-process the data according to preset rules, including data screening and prediction.

[0172] The analysis module 604 is used to perform algorithmic analysis on the processed data in multiple search spaces set according to different thresholds to obtain a set of root causes of abnormal points.

[0173] The output module 606 is used to sort the root cause set by frequency, and output the root cause whose frequency is greater than a preset value as the analysis result.

[0174] The specific functions of the above modules refer to the description in the above first embodiment and will not be repeated here.

[0175] The time series data anomaly root cause analysis system proposed in this embodiment is optimized based on the Squeeze algorithm, and provides a solution for automatically analyzing the possible causes of abnormal points in time series data. It can reduce a lot of manual analysis workload (saving time cost), and can also more efficiently and continuously output a standardized result (automatically output standardized data), facilitate query statistics and analysis (data storage and subsequent acquisition), and effectively improve the efficiency and accuracy of root cause analysis of abnormal points in time series data.

[0176] Embodiment 4

[0177] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores a timing data anomaly root cause analysis program, and the timing data anomaly root cause analysis program can be executed by at least one processor to enable the at least one processor to perform the steps of the timing data anomaly root cause analysis method as described above.

[0178] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0179] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0180] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0181] The above are only preferred embodiments of the embodiments of the present application, and are not intended to limit the patent scope of the embodiments of the present application. Any equivalent structure or equivalent process transformation made using the description and drawings of the embodiments of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the embodiments of the present application.

Claims

1. A method for analyzing root causes of abnormalities in time series data, characterized in that: The method comprises: Acquire the data to be analyzed from the database according to the input time information; Preprocessing the data according to preset rules, including data screening and prediction; Performing algorithmic analysis on the processed data in multiple search spaces set according to different thresholds to obtain a set of root causes of abnormal points; and The root cause set is sorted by frequency, and the root cause with a frequency greater than a preset value is output as an analysis result; The step of performing algorithmic analysis on the processed data in multiple search spaces set according to different thresholds to obtain a set of root causes of abnormal points includes: Converting the processed data into a data format required by a root cause analysis algorithm; Setting multiple possible values ​​of multiple thresholds required by the root cause analysis algorithm; Setting a plurality of search spaces according to the plurality of thresholds and the plurality of possible values; Performing a root cause analysis on the processed data using the root cause analysis algorithm in each search space to obtain a root cause analysis result; The root cause set is obtained by integrating the root cause analysis results of the multiple search spaces.

2. The method for analyzing root causes of abnormalities in time series data according to claim 1, characterized in that: The step of obtaining the data to be analyzed from the database according to the input time information comprises: Receive input time information; Verify the validity and format correctness of the time information; When the verification is passed, the data to be analyzed corresponding to the time information is obtained from the database.

3. The method for analyzing root causes of abnormalities in time series data according to claim 2, characterized in that: The step of obtaining the data to be analyzed corresponding to the time information from the database comprises: Set the dimensions of the data to be analyzed based on the business scenario; Data corresponding to the time information is obtained from the database according to the dimension.

4. The method for analyzing root causes of abnormalities in time series data according to claim 3, characterized in that: In the video playback business scenario, the dimensions include provinces, operators, and third-party content distribution network manufacturers.

5. The method for analyzing root causes of abnormalities in time series data according to any one of claims 1 to 4, characterized in that: The screening process includes: Set the key indicators and the weight conditions of the items corresponding to the key indicators; The data that does not meet the weight conditions will be eliminated.

6. The method for analyzing root causes of abnormalities in time series data according to claim 5, characterized in that: The key indicator is the playback freeze rate, the key indicator corresponding items are the number of video freezes and the number of video playbacks, and the weight condition is that the number of video playbacks is greater than a first threshold and the number of video freezes is greater than a second threshold.

7. The method for analyzing root causes of abnormalities in time series data according to claim 6, characterized in that: The screening process also includes: Set time series completeness conditions; Data that does not meet the completeness condition will be eliminated.

8. The method for analyzing root causes of abnormalities in time series data according to claim 7, characterized in that: The completeness condition includes: there is no missing data in the analysis point and the true value is not 0.

9. The method for analyzing root causes of abnormalities in time series data according to claim 8, characterized in that: The completeness condition also includes: the predicted value of the analysis point is greater than a third threshold and less than a fourth threshold.

10. The method for analyzing root causes of abnormalities in time series data according to any one of claims 1 to 4, 6, 8, and 9, characterized in that: The prediction process includes: Setting a plurality of time series prediction algorithms and a weight of each of the time series prediction algorithms; Get the true value of time series data in each dimension; Using the multiple time series prediction algorithms to predict the true values ​​respectively to generate corresponding predicted values; The predicted value and the weight obtained by each of the time series prediction algorithms are combined to calculate the final predicted value.

11. The method for analyzing root causes of abnormalities in time series data according to claim 1, characterized in that: The method further comprises: When performing root cause analysis on the processed data, the number of clustering intervals is adjusted according to the data volume of the business scenario.

12. The method for analyzing root causes of abnormalities in time series data according to claim 1, characterized in that: The root cause analysis algorithm is the Squeeze algorithm.

13. A time series data anomaly root cause analysis system, characterized in that: The system comprises: An acquisition module is used to acquire the data to be analyzed from the database according to the input time information; A processing module, used for preprocessing the data according to preset rules, including data screening and prediction; An analysis module is used to perform algorithmic analysis on the processed data in multiple search spaces set according to different thresholds to obtain a set of root causes of abnormal points; An output module, used to sort the root cause set by frequency, and output the root cause whose frequency is greater than a preset value as an analysis result; The step of performing algorithmic analysis on the processed data in multiple search spaces set according to different thresholds to obtain a set of root causes of abnormal points includes: Converting the processed data into a data format required by a root cause analysis algorithm; Setting multiple possible values ​​of multiple thresholds required by the root cause analysis algorithm; Setting a plurality of search spaces according to the plurality of thresholds and the plurality of possible values; Performing a root cause analysis on the processed data using the root cause analysis algorithm in each search space to obtain a root cause analysis result; The root cause set is obtained by integrating the root cause analysis results of the multiple search spaces.

14. An electronic device, characterized in that: The electronic device includes: a memory, a processor, and a time series data anomaly root cause analysis program stored in the memory and executable on the processor, wherein the time series data anomaly root cause analysis program implements the time series data anomaly root cause analysis method according to any one of claims 1 to 12 when executed by the processor.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a time series data anomaly root cause analysis program, and when the time series data anomaly root cause analysis program is executed by a processor, the time series data anomaly root cause analysis method according to any one of claims 1 to 12 is implemented.

16. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method described in any one of claims 1 to 12 are implemented.

Images