Dynamic monitoring method, system and equipment for multiple types of indexes and storage medium

Abstract

The embodiment of the invention discloses a dynamic monitoring method, system and device for multiple types of indexes and a storage medium. According to the technical scheme provided by the embodiment of the invention, the index statistical characteristics and the algorithm evaluation characteristics are acquired through the historical data corresponding to different types of indexes, and the pre-constructed index algorithm template library is queried based on the index statistical characteristics and the algorithm evaluation characteristics; and determining an index template matched with the index statistical characteristics and the algorithm evaluation characteristics as a target template, extracting a monitoring algorithm from the target template as a second monitoring algorithm, and performing index monitoring of the target type index in the next specified time period based on the second monitoring algorithm. By adopting the technical means, monitoring algorithm intelligent scheduling and dynamic monitoring can be carried out for different types of indexes, the situation that the adjusting process of the index monitoring algorithm is tedious and complex is avoided, the monitoring efficiency and monitoring accuracy of the different types of indexes are improved, the monitoring stability and reliability of the different types of indexes are improved, and the index monitoring effect is optimized.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a method, system, device and storage medium for dynamically monitoring multiple types of indicators. Background Art

[0002] At present, with the rapid development of the Internet industry and the increasing scale of Internet applications, more and more business systems generate a variety of indicator data. For example, the video freeze rate of the live broadcast business, the number of online people in the live broadcast room, the CPU of the service machine, memory usage, network latency, the number of request exceptions and other types of indicators need to be monitored and abnormal detection is performed on these business indicators, performance indicators, basic indicators, etc., in order to ensure the quality of the service, improve the efficiency of operation and maintenance, and avoid the loss of revenue.

[0003] However, related indicator monitoring solutions usually use the method of manually configuring indicator alarm thresholds, which triggers indicator abnormality alarms when the trigger conditions are met. Since there are many types of indicators that need to be monitored, it takes a lot of manpower to configure indicators and adjust thresholds. The whole process is cumbersome and complicated, and the monitoring rules are rigid. When the indicator form changes, it is easy to cause false alarms, resulting in poor indicator monitoring results. Summary of the invention

[0004] The embodiments of the present application provide a dynamic monitoring method, system, device and storage medium for multiple types of indicators, which can perform algorithm intelligent scheduling and dynamic monitoring for different types of indicators, improve the monitoring efficiency and monitoring accuracy of different types of indicators, and solve the cumbersome and complex technical problems in the monitoring process of different types of indicators.

[0005] In a first aspect, an embodiment of the present application provides a method for dynamically monitoring multiple types of indicators, including:

[0006] Obtain historical data of the target type indicator in a specified historical period, determine the algorithm monitoring results obtained by monitoring the indicator based on the historical data based on the first monitoring algorithm, determine the indicator statistical characteristics based on the historical data, and determine the algorithm evaluation characteristics based on the algorithm monitoring results, the indicator statistical characteristics are used to characterize the statistical results of the historical data, and the algorithm evaluation characteristics are used to characterize the monitoring effect of the first monitoring algorithm;

[0007] Based on the indicator statistical characteristics and algorithm evaluation characteristics, a pre-built indicator algorithm template library is queried to determine the indicator template that matches the indicator statistical characteristics and algorithm evaluation characteristics as the target template, and a monitoring algorithm is extracted from the target template as the second monitoring algorithm. The indicator algorithm template library pre-builds a variety of indicator templates corresponding to different types of indicators. The indicator template includes the corresponding monitoring algorithm and template characteristics for comparing the indicator statistical characteristics and algorithm evaluation characteristics.

[0008] The indicator monitoring of the target type indicator in the next specified time period is performed based on the second monitoring algorithm.

[0009] In a second aspect, an embodiment of the present application provides a dynamic monitoring system for multiple types of indicators, including:

[0010] a feature collection module configured to obtain historical data of a target type indicator in a specified historical period, determine an algorithm monitoring result obtained by monitoring the indicator based on the historical data using a first monitoring algorithm, determine a statistical feature of the indicator based on the historical data, and determine an algorithm evaluation feature based on the algorithm monitoring result, wherein the statistical feature of the indicator is used to characterize the statistical result of the historical data, and the algorithm evaluation feature is used to characterize the monitoring effect of the first monitoring algorithm;

[0011] The algorithm matching module is configured to query a pre-constructed indicator algorithm template library based on the indicator statistical characteristics and the algorithm evaluation characteristics, determine the indicator template that matches the indicator statistical characteristics and the algorithm evaluation characteristics as the target template, extract the monitoring algorithm from the target template as the second monitoring algorithm, the indicator algorithm template library pre-constructs a variety of indicator templates corresponding to different types of indicators, and the indicator template includes the corresponding monitoring algorithm and the template characteristics for comparing the indicator statistical characteristics and the algorithm evaluation characteristics;

[0012] The monitoring module is configured to perform indicator monitoring of the target type indicator in the next specified time period based on the second monitoring algorithm.

[0013] In a third aspect, an embodiment of the present application provides a dynamic monitoring device for multiple types of indicators, including:

[0014] memory and one or more processors;

[0015] The memory is configured to store one or more programs;

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the dynamic monitoring method of multiple types of indicators as described in the first aspect.

[0017] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions, when executed by a computer processor, are configured to execute the dynamic monitoring method of multiple types of indicators as described in the first aspect.

[0018] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes instructions. When the instructions are executed on a computer or a processor, the computer or the processor executes the dynamic monitoring method of multiple types of indicators as described in the first aspect.

[0019] The embodiment of the present application obtains the historical data of the target type indicator in the historical specified period, determines the algorithm monitoring result obtained by monitoring the indicator based on the first monitoring algorithm for the historical data, determines the indicator statistical characteristics based on the historical data, and determines the algorithm evaluation characteristics based on the algorithm monitoring result, the indicator statistical characteristics are used to characterize the statistical results of the historical data, and the algorithm evaluation characteristics are used to characterize the monitoring effect of the first monitoring algorithm; based on the indicator statistical characteristics and the algorithm evaluation characteristics, the pre-constructed indicator algorithm template library is queried, and the indicator template matching the indicator statistical characteristics and the algorithm evaluation characteristics is determined as the target template, and the monitoring algorithm is extracted from the target template as the second monitoring algorithm, and the indicator algorithm template library pre-constructs a plurality of indicator templates corresponding to different types of indicators, and the indicator template includes the corresponding monitoring algorithm and the template characteristics for comparing the indicator statistical characteristics and the algorithm evaluation characteristics; and the indicator monitoring of the target type indicator in the next specified period is performed based on the second monitoring algorithm. The above technical means are adopted to collect the indicator statistical characteristics and the algorithm evaluation characteristics by the historical data corresponding to the different types of indicators, determine the pre-set second monitoring algorithm based on the indicator statistical characteristics and the algorithm evaluation characteristics, and then perform the corresponding type of indicator monitoring based on the second monitoring algorithm. In this way, monitoring algorithms can be intelligently scheduled and dynamically monitored for different types of indicators, avoiding the cumbersome and complicated adjustment process of indicator monitoring algorithms, improving the monitoring efficiency and accuracy of different types of indicators, improving the stability and reliability of monitoring of different types of indicators, and optimizing indicator monitoring effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flow chart of a method for dynamic monitoring of multiple types of indicators provided in an embodiment of the present application;

[0021] Figure 2 It is an indicator monitoring flow chart in an embodiment of the present application;

[0022] Figure 3 This is a schematic diagram of the monitoring algorithm scheduling in an embodiment of the present application;

[0023] Figure 4 This is a schematic diagram of indicator merging alarm in an embodiment of the present application;

[0024] Figure 5 It is a structural diagram of a dynamic monitoring system for multiple types of indicators provided in an embodiment of the present application;

[0025] Figure 6 It is a structural diagram of a dynamic monitoring device for multiple types of indicators provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only the part related to the present application but not all the contents are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0027] The dynamic monitoring method for multiple types of indicators provided in the present application aims to collect indicator statistical characteristics and algorithm evaluation characteristics through historical data corresponding to different types of indicators, determine a pre-set second monitoring algorithm based on the indicator statistical characteristics and algorithm evaluation characteristics, and then perform corresponding type of indicator monitoring based on the second monitoring algorithm, so as to realize intelligent scheduling and dynamic monitoring of monitoring algorithms for different types of indicators.

[0028] With the rapid development of the Internet industry, the scale of Internet companies is growing, and more and more business systems generate a variety of indicator data, such as the video freeze rate of live broadcast services, the number of online people in the live broadcast room, the CPU of the machine where the service is located, memory usage, network latency, and the number of request exceptions. Internet application business operations need to monitor these business indicators, performance indicators, basic indicators, etc., and perform anomaly detection on them to ensure the quality of services, improve the efficiency of operation and maintenance, and avoid loss of revenue. However, most companies still use the method of manually configuring indicator alarm thresholds to monitor indicators, and send indicator abnormality alarms when the trigger conditions are met. This method requires a lot of manpower costs to configure and adjust thresholds for different indicators, and it is difficult to achieve a good monitoring effect.

[0029] Since there are a large number of indicators to be monitored and there are many types of indicators, most monitoring methods are limited to a certain indicator type or business scenario, and are highly customized, making it difficult to cover many complex types of indicator data. At the same time, in the face of many anomaly detection algorithms, what kind of indicator should be used for what kind of algorithm is mostly solved by engineers analyzing and solidifying the rules. The system relies heavily on human analysis results and lacks intelligence. In addition, due to the solidification of rules, when the indicator form changes, the built-in monitoring rules of the system may no longer apply, which will cause false alarms, thereby introducing a large amount of manual intervention costs.

[0030] Based on this, a dynamic monitoring method for multiple types of indicators is provided in an embodiment of the present application to intelligently schedule and dynamically monitor multiple monitoring algorithms for different types of indicators, thereby solving the cumbersome and complex technical problems of the monitoring process of different types of indicators.

[0031] Example:

[0032] Figure 1 A flowchart of a dynamic monitoring method for multiple types of indicators provided in an embodiment of the present application is provided. The dynamic monitoring method for multiple types of indicators provided in this embodiment can be executed by a dynamic monitoring device for multiple types of indicators. The dynamic monitoring device for multiple types of indicators can be implemented by software and / or hardware. The dynamic monitoring device for multiple types of indicators can be composed of two or more physical entities, or can be composed of one physical entity. Generally speaking, the dynamic monitoring device for multiple types of indicators can be a computing device such as a background server or a computer host.

[0033] The following description is made by taking the dynamic monitoring device of multiple types of indicators as the main body of the dynamic monitoring method of multiple types of indicators as an example. Figure 1 , the dynamic monitoring method of the multi-type indicators specifically includes:

[0034] S110. Obtain historical data of target type indicators in a specified historical period, determine the algorithm monitoring results obtained by monitoring the historical data based on the first monitoring algorithm, determine the indicator statistical characteristics based on the historical data, and determine the algorithm evaluation characteristics based on the algorithm monitoring results. The indicator statistical characteristics are used to characterize the statistical results of the historical data, and the algorithm evaluation characteristics are used to characterize the monitoring effect of the first monitoring algorithm.

[0035] When performing dynamic monitoring of different types of indicators (such as video freeze rate of live broadcast services, number of online users in the live broadcast room, memory usage, network latency, number of request exceptions, etc.), this application automatically selects the monitoring algorithm that best suits the current indicator characteristics in an intelligent manner to improve monitoring efficiency and accuracy and reduce human intervention and configuration costs.

[0036] Define the currently monitored indicator as a target type indicator. When selecting the monitoring algorithm for the current target type indicator, you first need to collect detailed data of the target type indicator in the specified historical period (such as the past week, month, etc.). These data can serve as the basis for subsequent analysis and algorithm selection. Ensure the accuracy and completeness of the data by cleaning the collected historical data and removing outliers and missing values.

[0037] It is understandable that at the initial moment of monitoring target type indicators, a basic or default monitoring algorithm will be selected to monitor the target type indicators in real time, detect anomalies and issue anomaly alarms. Figure 2In the subsequent target type indicator monitoring process, the indicator statistical characteristics and algorithm evaluation characteristics can be calculated based on the historical data and the algorithm monitoring results obtained by the indicator monitoring based on the monitoring algorithm of the historical data, and the intelligent routing scheduling of the monitoring algorithm used by the subsequent real-time data can be performed. The monitoring algorithm scheduled in real time is used for abnormal detection and alarm. The alarm results can be further fed back to the algorithm routing module by manual marking to assist in the next monitoring algorithm routing scheduling. The monitoring algorithm used by the current historical data is defined as the first monitoring algorithm, and the monitoring algorithm obtained by the subsequent intelligent scheduling is defined as the second monitoring algorithm.

[0038] Based on the historical data, a series of indicator statistical features can be calculated and extracted, and these features are used to characterize the statistical characteristics of the historical data. For the monitoring results according to the first monitoring algorithm, its monitoring effect is evaluated, and corresponding features are extracted as algorithm evaluation features to characterize the monitoring performance of the algorithm.

[0039] The statistical characteristics of indicators include at least one of the specified basic characteristics of historical data (such as mean, standard deviation, maximum value, minimum value, rate of change, etc.), distribution characteristics, autocorrelation characteristics and frequency domain characteristics, and the algorithm evaluation characteristics include at least one of prediction error, warning rate and warning accuracy.

[0040] Among them, this application collects historical data of no less than m days, and the interval of collected data points is de lta minutes, m = 14, de lta = 1. The statistical feature calculation of indicators is implemented using the Python time series feature extraction library tsfeature. The statistical features of indicators mainly include basic statistical features, distribution features, autocorrelation features and frequency domain features.

[0041] For the algorithm evaluation features, when determining the prediction error, we first calculate the mean absolute percentage error (MAPE) between the predicted value and the true value of the monitoring algorithm. For the convenience of observation, the indicator is converted into the Apdex indicator. The Apdex level is defined as follows in Table 1:

[0042] Table 1: Apdex level definitions

[0043] Apdex Level apdex scope excellent apdex>0.94 good 0.85≤apdex<0.94 generally 0.7≤apdex<0.85 Oops 0.5≤apdex<0.7 Unacceptable apdex<0.5

[0044] The Apdex index obtained above is used to characterize the prediction error.

[0045] The alarm rate alarm_rate refers to the proportion of alarm points in the observed historical data after being detected by the predefined detection algorithm and processed by the alarm rules. In theory, the alarm rate of the newly received indicators is mostly 0.

[0046]

[0047] The alarm accuracy rate alarm_accuracy refers to the proportion of correct alarms generated during the indicator detection process. This value can be calculated during system operation by users continuously labeling and feeding back the monitoring results.

[0048]

[0049] In actual applications, the above features and indicators can be selected and applied according to specific monitoring needs and business scenarios. For example, for scenarios that require highly accurate predictions, you can focus on prediction errors and alarm accuracy; while for scenarios that require rapid response, you pay more attention to alarm rates and the real-time nature of the algorithm.

[0050] At the same time, the calculation and analysis of the above characteristics and indicators can be carried out with the help of corresponding statistical methods and data processing tools, such as Python's Pandas library for data processing and statistical analysis, NumPy library for mathematical operations, and Matloplib, Seaborn and other libraries for data visualization. For the autocorrelation and frequency domain feature analysis of time series data, relevant functions and tools in libraries such as statsmode ls and scipy can also be used. This application does not impose fixed restrictions on the collection process of indicator statistical features and algorithm evaluation features, and will not be elaborated here.

[0051] S120. Query a pre-constructed indicator algorithm template library based on the indicator statistical characteristics and algorithm evaluation characteristics, determine the indicator template that matches the indicator statistical characteristics and algorithm evaluation characteristics as the target template, extract the monitoring algorithm from the target template as the second monitoring algorithm, the indicator algorithm template library pre-constructs a variety of indicator templates corresponding to different types of indicators, and the indicator template includes the corresponding monitoring algorithm and the template features for comparing the indicator statistical characteristics and the algorithm evaluation characteristics.

[0052] Furthermore, based on the above-determined indicator statistical features and algorithm evaluation features, a pre-built indicator algorithm template library is queried to determine a second monitoring algorithm for subsequent target type indicator monitoring. Prior to this, a template library containing multiple types of indicators and their corresponding multiple monitoring algorithms is pre-built. Each indicator template contains a specific monitoring algorithm and template features for comparing indicator statistical features and algorithm evaluation features.

[0053] Compare the calculated indicator statistical features and algorithm evaluation features with the template features in the indicator algorithm template library to find the best matching indicator template as the target template. Extract the corresponding monitoring algorithm from the target template as the second monitoring algorithm, which is the best monitoring algorithm under the current indicator characteristics and monitoring requirements.

[0054] Specifically, refer to Figure 3The indicator statistical features and algorithm evaluation features are combined into the initial composite features of the indicator, which are used as the input of the algorithm router. Optionally, the algorithm monitoring results obtained based on the monitoring algorithm can also be manually labeled to generate corresponding user feedback tags, which are included as the input of the algorithm router to assist in the selection of the second monitoring algorithm. The algorithm router consists of two layers, the first layer is the algorithm selector, and the second layer is the algorithm parameter selector.

[0055] Among them, the algorithm selector mainly selects the appropriate algorithm based on the composite characteristics of the indicator. Different types of indicator templates, as well as their suitable algorithm types and parameters, are built into the system. The indicator template can be manually pre-set according to the actual business monitoring needs. In the subsequent system operation process, the evaluation indicators are evaluated and the template is automatically formed after the indicator with a high alarm accuracy reaches a certain threshold. Each time a new indicator is added, the characteristics of the new indicator are matched with the characteristics of the indicator template, and the matched indicator template is determined as the target template.

[0056] Optionally, determining an indicator template that matches the indicator statistical characteristics and the algorithm evaluation characteristics as a target template includes:

[0057] Calculate the similarity distance between the indicator statistical features and algorithm evaluation features and the template features of each indicator template, and select the indicator template with the highest similarity distance as the target template.

[0058] The similarity calculation method can be selected according to the properties of the specific features. The similarity calculation methods include Euclidean distance, cosine similarity, Manhattan distance, and Jaccard similarity.

[0059] Since indicator features and template features may contain different types of data (such as numerical and categorical types), when there are multiple types of data, multiple similarity calculation methods can be used in combination, or different types of features can be preprocessed and then the similarity can be calculated uniformly. After calculating the similarity between all templates and the given feature, the indicator template with the highest similarity is selected as the target template. Among them, by defining and extracting the template features of all indicator templates, the determined indicator statistical features and algorithm evaluation features are extracted. Then, the similarity calculation method pre-specified for the current indicator type is selected, and the similarity between the indicator statistical features and algorithm evaluation features and each template feature is calculated. According to the similarity sorting, the template with the highest similarity is selected as the target template.

[0060] Optionally, extracting a monitoring algorithm from the target template as a second monitoring algorithm includes:

[0061] When multiple monitoring algorithms are extracted from the target template, the extracted monitoring algorithms are combined and arranged into composite algorithms based on the Python script to obtain the corresponding second monitoring algorithm.

[0062] The second monitoring algorithm of the present application can be a general algorithm that has been implemented, or it can be a composite algorithm composed of a plurality of defined monitoring algorithms to achieve a customized detection method. For example, dynamic threshold detection and mutation detection are combined to detect extreme anomalies where mutations exceed the threshold. The combination of different algorithms is arranged through a python script, so that the second monitoring algorithm has better scalability. By parsing and extracting all monitoring algorithms from the target template, a function or class is written in Python in advance, which will contain the logic of the composite algorithm, and the python script can be used to arrange the composite algorithm to generate the second monitoring algorithm.

[0063] Furthermore, after selecting the monitoring algorithm based on the above algorithm selector, the monitoring algorithm used by the subsequent target type indicator and the initial algorithm parameters contained in the monitoring algorithm can be preliminarily determined, and then the parameters can be further fine-tuned through the second-level algorithm parameter selector.

[0064] After extracting the monitoring algorithm from the target template as the second monitoring algorithm, the method further includes:

[0065] Receive an artificial feedback label generated based on the algorithm monitoring result, and adjust the specified algorithm parameters of the second monitoring algorithm according to the artificial feedback label to adjust the monitoring effect of the second monitoring algorithm.

[0066] The artificial feedback label of the present application includes at least one of a false alarm determination label, a specified algorithm suppression label, an alarm sensitivity label and a custom label. The algorithm parameter selector mainly fine-tunes the parameters of the algorithm based on the label features of user feedback. The labels fed back by users are mainly divided into four categories. One is the false alarm judgment label that indicates whether the alarm is accurate or not. This label is used to calculate the alarm accuracy of the indicator and can evaluate the monitoring effect of the system. At the same time, this label can be used to determine whether the indicator and the corresponding algorithm can be included in the template library. One is the specified algorithm suppression label. Each algorithm in the system corresponds to a detected anomaly. Suppressing the detection of a certain type of anomaly is equivalent to suppressing the use of the corresponding algorithm, so the algorithm selector will not select the corresponding algorithm. One is the alarm sensitivity label, which acts on the algorithm parameter selector. For example, when the alarm sensitivity label feedback alarm is too sensitive, the system will automatically adjust the relevant parameters of the algorithm to reduce the sensitivity. Otherwise, the sensitivity will be increased by automatically adjusting the relevant parameters of the algorithm. The last one is the user-defined label. This type of label uses text type and can be integrated into the composite features of the above indicators. When there are many indicators of this type of label, the template indicator of this type of label can be formed according to the evaluation indicator. At the same time, developers can also analyze these labels offline to further speed up the system's adaptation to the label.

[0067] Based on the artificial feedback tags generated by the above users based on the algorithm monitoring results, the algorithm parameters can be fine-tuned adaptively. At the same time, some tags are incorporated into the algorithm selector, which can also assist in the decision-making of the selection of the second monitoring algorithm, so as to achieve more accurate and intelligent monitoring algorithm scheduling and dynamic adjustment.

[0068] Optionally, after determining the monitoring algorithm and parameters to be used for the target type indicator, if the monitoring algorithm needs to use a prediction algorithm to calculate the dynamic threshold, the calculation of the indicator threshold needs to be performed regularly.

[0069] The specific scheduling rules for calculating the indicator threshold are as follows:

[0070] The scheduler receives the following parameters: the threshold advance prediction duration is delta, the proportion of the prediction timeout time to the prediction duration is p, the number of benchmark points for each prediction is num, the number of historical data points used is N, and the time unit is minutes. For each indicator, each collection node collects a new data point x t , and then based on the collected historical data, the dynamic prediction of the indicator threshold is performed. The dynamic threshold can learn the change law of the indicator through the historical data of the indicator, automatically generate the threshold and dynamically adjust the threshold. Among them, the Prophet algorithm can be used to calculate the threshold. Prophet is a set of time series prediction tools open-sourced by Facebook, which combines fully automatic prediction and online learning to ensure that the tool can solve most business problems. The dynamic threshold can also be obtained through the smoothing processing of historical data. This application does not make a fixed limit on the specific method of threshold prediction and will not elaborate here.

[0071] Among them, when no indicator threshold for any time period is generated for the current corresponding indicator, an initialization time range [t, t + random(p×delta + 1, delta)] is generated at this time, so that in subsequent iterations, different indicators are triggered to predict the threshold one by one within the next 1 to delta time.

[0072] In the subsequent dynamic threshold prediction process, by reading the currently predicted time range of the indicator, assumed to be [t1, t2], the threshold within this time period already exists, or it is the initialization time range;

[0073] If t < t2 - delta, no prediction needs to be triggered;

[0074] If t ∈ [t2 - delta, t2 - p×delta], a new prediction range [t2 + 60, t2 + 60 + random(num, num + delta)] is triggered and replaces the original time range;

[0075] If t>t2-p×delta, the process will be re-initialized. It should be noted that this situation is caused by the interruption of collected data, and the system can be configured with corresponding alarms.

[0076] In practical applications, assuming that dynamic thresholds of n indicators and a concurrency of q / min are supported, n = (delta×q)×p, the algorithm routing can configure different scheduling parameters for different types of indicators, so that the dynamic threshold calculation can support more indicators without causing concurrency problems.

[0077] S130. Perform indicator monitoring of the target type indicator in the next specified time period based on the second monitoring algorithm.

[0078] After that, based on the second monitoring algorithm determined above, the selected second monitoring algorithm can be used to monitor the target type indicator in real time during the next specified period. During this process, the system continuously collects new monitoring data and evaluates the monitoring effect based on this data. If it is found that the monitoring effect has declined or the indicator characteristics have changed, the above process can be re-executed to select a new monitoring algorithm to achieve dynamic adjustment. And so on, to achieve efficient and accurate indicator monitoring.

[0079] In this way, the accuracy of indicator detection can be improved by arranging different algorithm combinations to meet different detection effects; after intelligent scheduling of different indicators, appropriate anomaly detection methods and parameters are selected to adapt to all types of indicators; at the same time, real-time detection effect evaluation indicators and manual feedback labels are used to promptly feedback to the scheduler, so as to achieve the effect of dynamically adjusting the algorithm and improving the detection accuracy with minimal manual intervention.

[0080] Optionally, the indicator monitoring of the target type indicator in the next specified time period based on the second monitoring algorithm includes:

[0081] In the case where the second monitoring algorithm includes multiple sub-algorithms, a combined alarm of the target type indicators is performed based on the specified monitoring results of each sub-algorithm.

[0082] Reference Figure 4 , because in the multi-model intelligent scheduling framework, multiple monitoring algorithms can be selected, it is possible to have multiple anomaly detection results at the same time, such as sudden increase anomaly, pattern change anomaly, and above threshold anomaly. Based on the monitoring anomaly of multiple seed algorithms, the anomaly needs to be integrated in the alarm module, and multiple anomalies at the same time need to be merged and issued. Figure 3 As shown, when outlier anomalies, mutation anomalies, and threshold comparison anomalies are detected at the same time, the anomalies are integrated and then the alarms are merged.

[0083] After receiving an alarm, the user can provide feedback on whether the alarm is a false alarm, so as to form a false alarm judgment label for abnormality or not, and form an abnormal point sample library to evaluate the alarm accuracy of the indicator; the user can also provide feedback that there is no need to detect a certain abnormality, so as to form a specified algorithm suppression label, so as to suppress it when selecting the second-layer algorithm of the algorithm router; the user can also provide feedback that the alarm is too sensitive, so as to form an alarm sensitivity label, so as to make adjustments when selecting the algorithm parameters of the algorithm router; the user can also provide feedback on custom labels to form a label library of indicators.

[0084] For example, after receiving an alarm, the user feedbacks the specified algorithm suppression label of "do not pay attention to outliers", and the system automatically removes the corresponding outlier detection model during the next detection and no longer performs the corresponding abnormal detection.

[0085] Exemplarily, based on the above-mentioned indicator monitoring method, when monitoring indicators, users configure the data source and the query statement of the indicator through the UI interface to collect the indicators that need to be detected regularly. Among them, the configuration of the statement supports multiple time series data storage such as MySQL, VictoriaMetri cs, and ClickHouse. The query statement query obtains a multi-dimensional indicator. The collection statement of the single-dimensional indicator is obtained by parsing the query statement, and the historical data of each single-dimensional indicator is collected and cached for subsequent threshold prediction. At the same time, through the scheduled task control, the real-time data of all single-dimensional indicators is collected every 1 minute for abnormal monitoring and updating the historical data cache. In addition, for indicators that do not meet expectations, filtering must be completed and will not enter the next link. After collecting the data, the data of the last 14 days is stored in redis for easy reading; the historical data is stored in clickHouse for persistent storage.

[0086] Then, based on the collected historical data of the corresponding indicator, the statistical characteristics of the indicator are calculated, and the indicator evaluation characteristics of the indicator monitoring results of the historical data are pre-generated. The indicator statistical characteristics and indicator evaluation characteristics are sent to the algorithm router for algorithm routing, and the appropriate monitoring algorithm and parameters are selected to achieve the effect of dynamically adjusting the indicator algorithm. Then, the indicator monitoring is performed based on the determined monitoring algorithm and parameters. Based on the alarm parameters configured by the user from the UI interface, including the threshold comparison rules, if the indicator exceeds the upper limit or is lower than the lower limit abnormally, the alarm rule is triggered. Among them, an alarm is triggered if the above abnormal situation occurs several times in a row, and for alarms with the same abnormal type at the same time, the notification level is merged.

[0087] In the above, by obtaining the historical data of the target type indicator in the specified historical period, determining the algorithm monitoring result obtained by monitoring the indicator based on the first monitoring algorithm based on the historical data, determining the indicator statistical characteristics based on the historical data, and determining the algorithm evaluation characteristics based on the algorithm monitoring result, the indicator statistical characteristics are used to characterize the statistical results of the historical data, and the algorithm evaluation characteristics are used to characterize the monitoring effect of the first monitoring algorithm; querying the pre-constructed indicator algorithm template library based on the indicator statistical characteristics and the algorithm evaluation characteristics, determining the indicator template matching the indicator statistical characteristics and the algorithm evaluation characteristics as the target template, extracting the monitoring algorithm from the target template as the second monitoring algorithm, the indicator algorithm template library pre-constructs a variety of indicator templates corresponding to different types of indicators, the indicator template contains the corresponding monitoring algorithm and the template characteristics for comparing the indicator statistical characteristics and the algorithm evaluation characteristics; and performing the indicator monitoring of the target type indicator in the next specified period based on the second monitoring algorithm. By adopting the above technical means, the indicator statistical characteristics and the algorithm evaluation characteristics are collected by the historical data corresponding to different types of indicators, the pre-set second monitoring algorithm is determined based on the indicator statistical characteristics and the algorithm evaluation characteristics, and then the corresponding type of indicator monitoring is performed based on the second monitoring algorithm. In this way, monitoring algorithms can be intelligently scheduled and dynamically monitored for different types of indicators, avoiding the cumbersome and complicated adjustment process of indicator monitoring algorithms, improving the monitoring efficiency and accuracy of different types of indicators, improving the stability and reliability of monitoring of different types of indicators, and optimizing indicator monitoring effects.

[0088] Based on the above embodiments, Figure 5 A schematic diagram of the structure of a dynamic monitoring system for multiple types of indicators provided by this application. Figure 5 The dynamic monitoring system for multiple types of indicators provided in this embodiment specifically includes: a feature collection module 21, an algorithm matching module 22 and a monitoring module 23.

[0089] Among them, the feature collection module 21 is configured to obtain historical data of the target type indicator in a specified historical period, determine the algorithm monitoring result obtained by monitoring the indicator based on the historical data based on the first monitoring algorithm, determine the indicator statistical characteristics based on the historical data, and determine the algorithm evaluation characteristics based on the algorithm monitoring result. The indicator statistical characteristics are used to characterize the statistical results of the historical data, and the algorithm evaluation characteristics are used to characterize the monitoring effect of the first monitoring algorithm;

[0090] The algorithm matching module 22 is configured to query a pre-constructed indicator algorithm template library based on the indicator statistical characteristics and the algorithm evaluation characteristics, determine the indicator template that matches the indicator statistical characteristics and the algorithm evaluation characteristics as the target template, extract the monitoring algorithm from the target template as the second monitoring algorithm, and the indicator algorithm template library pre-constructs a plurality of indicator templates corresponding to different types of indicators, and the indicator template includes the corresponding monitoring algorithm and the template characteristics for comparing the indicator statistical characteristics and the algorithm evaluation characteristics;

[0091] The monitoring module 23 is configured to perform indicator monitoring of the target type indicator in the next specified time period based on the second monitoring algorithm.

[0092] The indicator statistical characteristics include at least one of the specified basic characteristics, distribution characteristics, autocorrelation characteristics and frequency domain characteristics of historical data, and the algorithm evaluation characteristics include at least one of the prediction error, warning rate and warning accuracy rate.

[0093] Specifically, extracting a monitoring algorithm from the target template as the second monitoring algorithm includes:

[0094] When multiple monitoring algorithms are extracted from the target template, the extracted monitoring algorithms are combined and arranged into composite algorithms based on the Python script to obtain the corresponding second monitoring algorithm.

[0095] Specifically, an indicator template that matches the indicator statistical characteristics and algorithm evaluation characteristics is determined as a target template, including:

[0096] Calculate the similarity distance between the indicator statistical features and algorithm evaluation features and the template features of each indicator template, and select the indicator template with the highest similarity distance as the target template.

[0097] Specifically, after extracting the monitoring algorithm from the target template as the second monitoring algorithm, the method further includes:

[0098] Receive an artificial feedback label generated based on the algorithm monitoring result, and adjust the specified algorithm parameters of the second monitoring algorithm according to the artificial feedback label to adjust the monitoring effect of the second monitoring algorithm.

[0099] Specifically, the manual feedback label includes at least one of a false alarm determination label, a specified algorithm suppression label, an alarm sensitivity label, and a custom label.

[0100] Specifically, the indicator monitoring of the target type indicator in the next specified time period is performed based on the second monitoring algorithm, including:

[0101] In the case where the second monitoring algorithm includes multiple sub-algorithms, a combined alarm of the target type indicators is performed based on the specified monitoring results of each sub-algorithm.

[0102] In the above, by obtaining the historical data of the target type indicator in the specified historical period, determining the algorithm monitoring result obtained by monitoring the indicator based on the first monitoring algorithm based on the historical data, determining the indicator statistical characteristics based on the historical data, and determining the algorithm evaluation characteristics based on the algorithm monitoring result, the indicator statistical characteristics are used to characterize the statistical results of the historical data, and the algorithm evaluation characteristics are used to characterize the monitoring effect of the first monitoring algorithm; querying the pre-constructed indicator algorithm template library based on the indicator statistical characteristics and the algorithm evaluation characteristics, determining the indicator template matching the indicator statistical characteristics and the algorithm evaluation characteristics as the target template, extracting the monitoring algorithm from the target template as the second monitoring algorithm, the indicator algorithm template library pre-constructs a variety of indicator templates corresponding to different types of indicators, the indicator template contains the corresponding monitoring algorithm and the template characteristics for comparing the indicator statistical characteristics and the algorithm evaluation characteristics; and performing the indicator monitoring of the target type indicator in the next specified period based on the second monitoring algorithm. By adopting the above technical means, the indicator statistical characteristics and the algorithm evaluation characteristics are collected by the historical data corresponding to different types of indicators, the pre-set second monitoring algorithm is determined based on the indicator statistical characteristics and the algorithm evaluation characteristics, and then the corresponding type of indicator monitoring is performed based on the second monitoring algorithm. In this way, monitoring algorithms can be intelligently scheduled and dynamically monitored for different types of indicators, avoiding the cumbersome and complicated adjustment process of indicator monitoring algorithms, improving the monitoring efficiency and accuracy of different types of indicators, improving the stability and reliability of monitoring of different types of indicators, and optimizing indicator monitoring effects.

[0103] The dynamic monitoring system for multiple types of indicators provided in the embodiments of the present application can be configured to execute the dynamic monitoring method for multiple types of indicators provided in the above embodiments, and has corresponding functions and beneficial effects.

[0104] Based on the above practical example, the present application embodiment also provides a dynamic monitoring device for multiple types of indicators, referring to Figure 6, the dynamic monitoring device for multiple types of indicators includes: a processor 31, a memory 32, a communication module 33, an input device 34 and an output device 35. The memory, as a computer-readable storage medium, can be configured to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the dynamic monitoring method for multiple types of indicators described in any embodiment of the present application (for example, a feature acquisition module, an algorithm matching module and a monitoring module in a dynamic monitoring system for multiple types of indicators). The communication module is configured to perform data transmission. The processor executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory, that is, to realize the dynamic monitoring method for multiple types of indicators mentioned above. The input device can be configured to receive input digital or character information, and generate key signal input related to user settings and function control of the device. The output device may include a display device such as a display screen. The dynamic monitoring device for multiple types of indicators provided above can be configured to execute the dynamic monitoring method for multiple types of indicators provided in the above embodiment, and has corresponding functions and beneficial effects.

[0105] On the basis of the above embodiments, the embodiments of the present application further provide a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, wherein the computer-executable instructions are configured to execute a dynamic monitoring method for multiple types of indicators when executed by a computer processor, and the storage medium may be any of various types of memory devices or storage devices. Of course, the computer-readable storage medium provided in the embodiments of the present application, whose computer-executable instructions are not limited to the dynamic monitoring method for multiple types of indicators as described above, may also execute related operations in the dynamic monitoring method for multiple types of indicators provided in any embodiment of the present application.

[0106] On the basis of the above embodiments, the embodiments of the present application also provide a computer program product. The technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer program product is stored in a storage medium, and includes a number of instructions for enabling a computer device, a mobile terminal or a processor therein to execute all or part of the steps of the dynamic monitoring method of multiple types of indicators described in each embodiment of the present application.

Claims

1. A dynamic monitoring method for multiple types of indicators, characterized in that: include: Obtain historical data of the target type indicator in a specified historical period, determine an algorithm monitoring result obtained by monitoring the indicator based on the historical data using a first monitoring algorithm, determine indicator statistical features based on the historical data, and determine algorithm evaluation features based on the algorithm monitoring results, wherein the indicator statistical features are used to characterize the statistical results of the historical data, and the algorithm evaluation features are used to characterize the monitoring effect of the first monitoring algorithm; Based on the indicator statistical characteristics and the algorithm evaluation characteristics, a pre-constructed indicator algorithm template library is queried, an indicator template matching the indicator statistical characteristics and the algorithm evaluation characteristics is determined as a target template, and a monitoring algorithm is extracted from the target template as a second monitoring algorithm, wherein the indicator algorithm template library pre-constructs a plurality of indicator templates corresponding to different types of indicators, and the indicator template includes a corresponding monitoring algorithm and a template feature for comparing the indicator statistical characteristics and the algorithm evaluation characteristics; Based on the second monitoring algorithm, indicator monitoring of the target type indicator in the next specified time period is performed.

2. The dynamic monitoring method of multiple types of indicators according to claim 1 is characterized in that: The indicator statistical characteristics include at least one of the specified basic characteristics, distribution characteristics, autocorrelation characteristics and frequency domain characteristics of the historical data, and the algorithm evaluation characteristics include at least one of the prediction error, warning rate and warning accuracy rate.

3. The dynamic monitoring method of multiple types of indicators according to claim 1 is characterized in that: The extracting a monitoring algorithm from the target template as a second monitoring algorithm includes: In the case where multiple monitoring algorithms are extracted from the target template, the extracted monitoring algorithms are combined and arranged into composite algorithms based on a python script to obtain a corresponding second monitoring algorithm.

4. The method for dynamic monitoring of multiple types of indicators according to any one of claims 1 to 3, characterized in that: The determining an indicator template matching the indicator statistical feature and the algorithm evaluation feature as a target template includes: The similarity distances between the indicator statistical features and the algorithm evaluation features and the template features of each indicator template are calculated, and the indicator template with the highest similarity distance is selected as the target template.

5. The method for dynamic monitoring of multiple types of indicators according to any one of claims 1 to 3, characterized in that: After extracting the monitoring algorithm from the target template as the second monitoring algorithm, the method further includes: Receive an artificial feedback label generated based on the algorithm monitoring result, and adjust the specified algorithm parameters of the second monitoring algorithm according to the artificial feedback label to adjust the monitoring effect of the second monitoring algorithm.

6. The dynamic monitoring method of multiple types of indicators according to claim 5 is characterized in that: The manual feedback label includes at least one of a false alarm determination label, a specified algorithm suppression label, an alarm sensitivity label and a custom label.

7. The method for dynamic monitoring of multiple types of indicators according to any one of claims 1 to 3, characterized in that: The monitoring of the target type indicator in the next specified time period based on the second monitoring algorithm includes: In the case where the second monitoring algorithm includes multiple sub-algorithms, a combined alarm for the target type indicator is performed based on the specified monitoring results of each sub-algorithm.

8. A dynamic monitoring system for multiple types of indicators, characterized in that: include: a feature collection module configured to obtain historical data of a target type indicator in a specified historical period, determine an algorithm monitoring result obtained by monitoring the indicator based on the historical data using a first monitoring algorithm, determine an indicator statistical feature based on the historical data, and determine an algorithm evaluation feature based on the algorithm monitoring result, wherein the indicator statistical feature is used to characterize the statistical result of the historical data, and the algorithm evaluation feature is used to characterize the monitoring effect of the first monitoring algorithm; An algorithm matching module is configured to query a pre-constructed indicator algorithm template library based on the indicator statistical characteristics and the algorithm evaluation characteristics, determine an indicator template that matches the indicator statistical characteristics and the algorithm evaluation characteristics as a target template, and extract a monitoring algorithm from the target template as a second monitoring algorithm, wherein the indicator algorithm template library pre-constructs a plurality of indicator templates corresponding to different types of indicators, and the indicator template includes a corresponding monitoring algorithm and a template feature for comparing the indicator statistical characteristics and the algorithm evaluation characteristics; A monitoring module is configured to perform indicator monitoring of the target type indicator in the next specified time period based on the second monitoring algorithm.

9. A dynamic monitoring device for multiple types of indicators, characterized in that: include: memory and one or more processors; The memory is configured to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the dynamic monitoring method of multiple types of indicators as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a computer processor, are configured to execute the method for dynamically monitoring multiple types of indicators as described in any one of claims 1 to 7.

11. A computer program product, characterized in that The computer program product includes instructions, and when the instructions are executed on a computer or a processor, the computer or the processor executes the method for dynamically monitoring multiple types of indicators as described in any one of claims 1 to 7.

Images