A threshold recommendation method for adaptively reducing jitter alarms

Abstract

The present invention discloses a threshold recommendation method for adaptively reducing jitter alarms. The present invention can intelligently extract recommended thresholds from a model, and has sufficient flexibility, and can fine-tune the model structure according to the experience of operation and maintenance personnel and new business scenarios. For example, the model optimization direction can be fine-tuned according to the severity level of the alarm, the time taken from the alarm opening to the alarm closing, and whether to pay more attention to the noise reduction rate or the accuracy rate. It can also adapt to the situation of three or more threshold indicators according to the needs of the business scenario. In view of the unstable characteristics of alarm data changing over time, by saving the weights of the most recently trained model, incremental learning is directly based on the weights of the historically trained model when the latest data is trained next time, which effectively improves the adaptive ability of the threshold recommendation. At the same time, the present invention introduces a final splitting mechanism to ensure the availability of the model threshold recommendation results and ensure the stability and reliability of the system.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent operation and maintenance systems, and in particular to a threshold recommendation method for adaptively reducing jitter alarms. Background Art

[0002] With the rapid development of the Internet and the emergence of distributed systems, the scale of the backend of today's operation and maintenance systems is becoming more and more complex and large. In order to achieve overall management of the system, it is necessary to monitor the system in all aspects; for example, at the physical layer, it is necessary to monitor the use of various resources such as server CPU and memory; at the data layer, it is necessary to monitor indicators such as database load; at the application layer, it is necessary to monitor the performance and availability of applications. Large-scale operation and maintenance systems generate dozens or even hundreds of alarms per minute, among which jitter type alarms (that is, alarms are turned on and then closed within a short period of time, such as within one minute) are more common, and this large number of jitter type alarms will inevitably lead to the technical staff being unable to pay better attention to the alarms that really need attention. However, the manual configuration of monitoring thresholds has the disadvantages of being highly subjective or unable to be dynamically adjusted, resulting in many invalid alarms.

[0003] Therefore, the prior art has defects and needs to be improved. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a threshold recommendation method for adaptively reducing jitter alarms.

[0005] The technical solution of the present invention is as follows: a threshold recommendation method for adaptively reducing jitter alarms is provided, comprising the following steps:

[0006] Step 1: Read the existing alarm data of the user system from the database as the original data, pre-process the original data, disassemble the text, extract the specific values, and integrate the values ​​of the same business;

[0007] Step 2: Use the pre-processed raw data as the alarm data information for this execution process, perform rule screening, and obtain the current optimal score;

[0008] Step 3: Adjust the xgboost model structure based on the experience of the operation and maintenance personnel and the new business scenarios. Train the model based on the rule-filtered data combined with the feature data that does not need to be adjusted. Each time the xgboost model is executed, the model is segmented and a segmentation map is generated. The corresponding leaf score is calculated and stored based on the model evaluation indicator AUC, which is recorded as the model training score.

[0009] Step 4: Determine whether the model meets the threshold recommendation requirements based on the model training score and the current optimal score, and then determine whether it meets the actual business requirements. If it does not meet the threshold recommendation requirements, no processing is performed and the threshold remains unchanged; if it meets the threshold recommendation requirements, proceed to step 5;

[0010] Step 5: Based on the segmentation path and segmentation score of the historical model segmentation map, the current model segmentation map is combined for incremental learning to determine the final model segmentation map;

[0011] Step 6: Determine whether the final model segmentation map determined in step 5 meets the business requirements. If it does not meet the business requirements, no processing is performed and the threshold remains unchanged. If it meets the business requirements, proceed to step 7.

[0012] Step 7: Perform threshold analysis based on the final model segmentation map determined in step 5, and calculate the threshold recommendation confidence based on the segmentation score and the number of samples.

[0013] Furthermore, the rule screening of step 2 is to eliminate alarm categories that do not require threshold recommendations and threshold recommendations that cannot be trained by the rapid generation model according to business rules, and the rule screening method includes the following classifications: all are jitter alarms, none are jitter alarms, and non-jitter and jitter alarms overlap.

[0014] Furthermore, the adjustable model structure of the xgboost model in step 3 includes: weighing the evaluation index according to the noise reduction rate and the accuracy rate, weighing the category weight according to the time taken from the alarm being turned on to being turned off, and weighing the sample weight according to the severity level of the alarm.

[0015] Furthermore, the evaluation index based on the trade-off between noise reduction rate and accuracy is F β The noise reduction rate and accuracy are weighed and the segmentation score is calculated. The calculation formula is as follows:

[0016] in,

[0017] F β It is the calculation result of the custom F1 calculation formula;

[0018] β is the penalty term. When β=1, it is the standard F1 calculation formula. When β>1, the accuracy is penalized. When β<1, the denoising rate is penalized.

[0019] P is the noise reduction rate, and R is the accuracy rate.

[0020] Furthermore, the xgboost model in step 3 may be replaced by a catboost model or a lightgbm model.

[0021] Furthermore, the operation and maintenance personnel experience in step 3 provides relevant interactive pages for the front end based on the classification and summary of historical information.

[0022] Furthermore, the specific method of model segmentation is as follows: sort the training data according to each feature and save them into several block structures, the number of block structures is equal to the number of features; divide each feature from small to large, compare the size of the objective function after each division, and select the node with the largest decrease as the optimal division point of the feature; compare the decrease values ​​of the objective function of the optimal division points of different block structures, and select the feature with the largest decrease as the optimal division point, so as to obtain the segmentation score.

[0023] Furthermore, the incremental learning refers to injecting new data into a trained model within a set period, so that the model can relearn and update the data into the database.

[0024] By adopting the above scheme, the present invention can intelligently extract recommended thresholds from the model, and has sufficient flexibility. The model structure can be fine-tuned according to the experience of operation and maintenance personnel and new business scenarios. For example, the optimization direction of the model can be fine-tuned according to the severity level of the alarm, the time taken from the alarm opening to the alarm closing, and whether to pay more attention to the noise reduction rate or the accuracy rate. It can also adapt to the situation of three or more threshold indicators according to the needs of the business scenario. In view of the unstable characteristics of alarm data changing over time, by saving the weights of the most recently trained model, incremental learning is directly based on the weights of the historically trained model when training the latest data next time, which effectively improves the adaptive ability of the threshold recommendation. At the same time, the present invention introduces a final splitting mechanism to ensure the availability of the model threshold recommendation results and ensure the stability and reliability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is the overall architecture diagram of the present invention.

[0026] Figure 2 Flowchart of the model recommendation module.

[0027] Figure 3 Model segmentation diagram and incremental learning flowchart.

[0028] Figure 4 is the split path binary tree graph. DETAILED DESCRIPTION

[0029] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0030] See also Figure 1 The present invention provides a threshold recommendation method for adaptively reducing jitter alarms, comprising the following steps:

[0031] Step 1: Read the stock alarm data of the user system from the database as the original data, pre-process the original data, disassemble the text, extract the specific values, and integrate the values ​​of the same business. The original data is the stock alarm data received and saved in the enterprise's system, including historical alarm data and newly generated alarm data.

[0032] Step 2: Use the pre-processed raw data as the alarm data information for this execution process, perform rule screening, and obtain the current optimal score. Each service alarm can be divided into jitter alarm or non-jitter alarm, and is screened by user-defined rules to remove jitter alarms, and calculate the recommended threshold and confidence.

[0033] Step 3: Adjust the xgboost model structure based on the experience of the operation and maintenance personnel and the new business scenarios. Train the model based on the rule-filtered data combined with the feature data that does not need to be adjusted. Each time the xgboost model is executed, the model is segmented and a segmentation map is generated. The corresponding leaf score is calculated and stored based on the model evaluation indicator AUC, which is recorded as the model training score.

[0034] The operation and maintenance personnel experience provides relevant interactive pages for the front end based on the classification and summary of historical information. Assuming that the threshold recommendation for the success rate is 70%, but the operation and maintenance personnel experience summarized from historical information believes that the 95% success rate is more reliable as the threshold recommendation limit, then you can choose not to accept the threshold recommendation, and the front end can be fed back to the back-end algorithm.

[0035] The specific method of model segmentation is as follows: sort the training data according to each feature and save them into several block structures, where the number of block structures is equal to the number of features; segment each feature from small to large, compare the size of the objective function after each segmentation, and select the node with the largest decrease as the optimal segmentation point for the feature; compare the decrease values ​​of the objective function of the optimal segmentation points of different block structures, and select the feature with the largest decrease as the optimal segmentation point, so as to obtain the segmentation score.

[0036] Different users, or the same user at different time points and task scenarios, need to use different data and parameters. Therefore, when executing new business scenarios, users are required to modify and adjust the parameters to meet the calculation requirements of the recommended thresholds.

[0037] AUC is a model evaluation indicator in the field of machine learning, which is the area under the ROC curve. Since the ROC curve is generally above the straight line y=x, the value range of AUC is between 0.5 and 1. The closer AUC is to 1.0, the higher the authenticity of the detection method; when it is equal to 0.5, the authenticity is the lowest and there is no application value.

[0038] Step 4: Determine whether the model meets the threshold recommendation requirements based on the model training score and the current optimal score, and then determine whether it meets the actual business requirements. If the threshold recommendation requirements are not met, no processing is performed and the threshold remains unchanged; if the threshold recommendation requirements are met, proceed to step 5.

[0039] Step 5: Based on the segmentation path and segmentation score of the historical model segmentation map, incremental learning is performed in combination with the current model segmentation map to determine the final model segmentation map. Incremental learning refers to injecting new data into the trained model within a set period, so that the model relearns and updates the data into the database. In some embodiments, the incremental learning of the current model segmentation map is achieved by injecting new data into the trained model and running the model in a time period of one week.

[0040] Step 6: Determine whether the final model segmentation map determined in step 5 meets the business requirements. If it does not meet the business requirements, no processing is performed and the threshold remains unchanged; if it meets the business requirements, proceed to step 7.

[0041] Step 7: Perform threshold analysis based on the final model segmentation map determined in step 5, and calculate the threshold recommendation confidence based on the segmentation score and sample number. Based on the segmentation path and segmentation score of the final model segmentation map, use the quantile algorithm of the xgboost model to calculate and obtain the recommended threshold, and compare the recommended threshold with the sample number to determine the confidence.

[0042] In statistics, the confidence interval of a probability sample is an interval estimate of a population parameter of the sample. The confidence interval shows the degree to which the true value of the parameter has a certain probability of falling around the measurement result, also known as the confidence level. If the calculated recommended threshold is 90%, and the confidence interval at the confidence level of 0.95 is (85%, 95%), then there is a 95% chance that the true recommended threshold falls between 85% and 95%. Conversely, by comparing the calculated recommended threshold with the sample size, the percentage of the recommended threshold and the number of non-jitter alarms in the sample can be determined, thereby calculating the confidence level.

[0043] The present invention uses a tree model, and further trains alarm data through an xgboost model. The xgboost model has the characteristics of parallel processing, high speed, high flexibility of customizable optimization targets and evaluation functions, and business interpretability of the model recommendation threshold. Then, the recommendation rules and recommendation rule weights are intelligently extracted from the model, and the determination of the alarm threshold is more intelligent and portable. At the same time, the present invention can save the historical recommendation rule weights of the model. The basic strategy selected is: give a greater weight to the model recommendation rule weights of the more recent time, and then adjust the threshold dynamically through incremental learning and other technologies, which effectively improves the adaptive ability of the threshold recommendation.

[0044] See also Figure 2 ,The overall process of the threshold recommendation model is as follows:

[0045] Rule filtering:

[0046] According to business rules, exclude alarm categories that do not require threshold recommendations and threshold recommendations that cannot be trained by the rapid generation model. The rule screening methods include the following categories:

[0047] All jitter alarms: remove all jitter alarms and generate thresholds and confidence levels;

[0048] All are jitter alarms: no processing is performed and the threshold remains unchanged;

[0049] The non-jitter alarms and jitter alarms overlap, and the ratio is 1:2: All jitter alarms are removed to generate thresholds and confidence levels.

[0050] Tree model training:

[0051] Compared with the traditional manually determined threshold rules, the Xgboost model has the following advantages:

[0052] The regularization term prevents overfitting and improves the applicability of the threshold rule;

[0053] The threshold rules are interpretable and can measure feature importance based on weight, coverage, and gain;

[0054] The Xgboost model uses not only the first-order derivative but also the second-order derivative, making the loss more accurate.

[0055] Considering the case where the training data is sparse, the default direction of the branch can be specified for missing values ​​or specified values, which can greatly improve the efficiency of the algorithm;

[0056] Support column sampling, which can not only reduce overfitting but also reduce calculation;

[0057] Engineering optimization, parallel column block design, cache access, and out-of-core fast computing have improved algorithm efficiency and disk throughput.

[0058] Fine-tune the model structure based on the experience of the operation and maintenance personnel and new business scenarios, and train the model based on the data filtered by the rules and other feature data; the adjustable model structures selected by the present invention are:

[0059] 1. Weigh the evaluation indicators based on noise reduction rate and accuracy:

[0060] In some applications, the degree of emphasis on precision and recall is different. For example, in a product recommendation system, in order to disturb users as little as possible, it is more important to recommend content that users are interested in. In this case, precision is more important; in a fugitive information retrieval system, it is more necessary to avoid omissions as much as possible, so recall is more important. Evaluation indicator F β Considering the direct relationship between the two, F β The noise reduction rate and accuracy are weighed, and the calculation formula is as follows:

[0061]

[0062] in,

[0063] F β It is the calculation result of the custom F1 calculation formula;

[0064] β is the penalty term. When β=1, it is the standard F1 calculation formula. When β>1, the accuracy is penalized. When β<1, the denoising rate is penalized.

[0065] P is the noise reduction rate, and R is the accuracy rate.

[0066] Among them, β>0 measures the relative importance of recall rate to precision rate, β=1 is the standard F1; β>1 has a greater impact on recall rate; β<1 has a greater impact on precision rate. By calculating the evaluation index F β , thus obtaining the segmentation score of the current model segmentation map.

[0067] 2. Weigh the category weights based on the time it takes for the alarm to be turned on and off.

[0068] 3. Weight the samples according to the severity of the alarm.

[0069] The optimal score is determined by comparing the model training score with the current score to ensure that the model training result is no worse than the result recommended by the previous manual threshold.

[0070] In some embodiments, a tree model such as catboost or lightgbm may be used instead of the xgboost model for calculation, or the recommendation threshold may be indirectly calculated by logistic regression or support vector machine in the following manner. Taking support vector machine as an example, the calculation method is as follows:

[0071] 1. Specify the kernel function of the support vector machine as a linear kernel function;

[0072] 2. Train the support vector machine model to obtain the explanatory variable coefficients and intercept terms of the support vector machine linear kernel function;

[0073] 3. Perform Bayesian search on the linear kernel function to obtain the optimal point to decompose the linear kernel function into a horizontal line A perpendicular to the x-axis and a horizontal line Y perpendicular to the y-axis;

[0074] 4. The obtained lines A and B are the recommended optional thresholds (similar to the case of three dimensions and above).

[0075] See also Figure 3 ,The specific process of model segmentation module and incremental learning is as follows:

[0076] 1. Extract all segmentation paths and segmentation scores from the tree model, as shown in the following table:

[0077]

[0078] Use a binary tree representation, such as Figure 4 shown.

[0079] 2. Save the segmentation path and segmentation score of this model, and merge the historical and current segmentation paths according to the segmentation score and historical weight;

[0080] 3. Determine the threshold analysis result based on the segmentation path and segmentation score;

[0081] 4. Determine the confidence level based on the segmentation score and sample size.

[0082] The adjustable model structure of the xgboost model in step 3 includes: weighing the evaluation index according to the noise reduction rate and the accuracy rate, weighing the category weight according to the time taken from the alarm being turned on to being turned off, and weighing the sample weight according to the severity level of the alarm.

[0083] In summary, the present invention can intelligently extract recommended thresholds from the model, and has sufficient flexibility. The model structure can be fine-tuned according to the experience of operation and maintenance personnel and new business scenarios. For example, the optimization direction of the model can be fine-tuned according to the severity level of the alarm, the time taken from the alarm opening to the alarm closing, and whether to pay more attention to the noise reduction rate or the accuracy rate. It can also adapt to three or more threshold indicators according to the needs of the business scenario. In view of the unstable nature of alarm data changing over time, by saving the weights of the most recently trained model, incremental learning is directly based on the weights of the historically trained model when training the latest data next time, which effectively improves the adaptive ability of the threshold recommendation. At the same time, the present invention introduces a final splitting mechanism to ensure the availability of the model threshold recommendation results and ensure the stability and reliability of the system.

[0084] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A threshold recommendation method for adaptively reducing jitter alarms, characterized in that: The steps include: Step 1: Read the existing alarm data of the user system from the database as the original data, pre-process the original data, disassemble the text, extract the specific values, and integrate the values ​​of the same business; Step 2: Use the pre-processed raw data as the alarm data information for this execution process, perform rule screening, and obtain the current optimal score; Step 3: Adjust the xgboost model structure based on the experience of the operation and maintenance personnel and the new business scenarios. Train the model based on the rule-filtered data combined with the feature data that does not need to be adjusted. Each time the xgboost model is executed, the model is segmented and a segmentation map is generated. The corresponding leaf score is calculated and stored based on the model evaluation indicator AUC, which is recorded as the model training score. Step 4: Determine whether the model meets the threshold recommendation requirements based on the model training score and the current optimal score, and then determine whether it meets the actual business requirements. If it does not meet the threshold recommendation requirements, no processing is performed and the threshold remains unchanged; If the threshold recommendation requirement is met, proceed to step 5; Step 5: Based on the segmentation path and segmentation score of the historical model segmentation map, the current model segmentation map is combined for incremental learning to determine the final model segmentation map; Step 6: Determine whether the final model segmentation map determined in step 5 meets the business requirements. If it does not meet the business requirements, no processing is performed and the threshold remains unchanged; If the business requirements are met, proceed to step 7; Step 7: Perform threshold analysis based on the final model segmentation map determined in step 5, and calculate the threshold recommendation confidence based on the segmentation score and the number of samples.

2. The threshold recommendation method for adaptively reducing jitter alarm according to claim 1, characterized in that: The rule screening of step 2 is to eliminate the alarm categories that do not require threshold recommendations and the threshold recommendations that cannot be trained by the rapid generation model according to business rules. The rule screening method includes the following classifications: all are jitter alarms, none are jitter alarms, and non-jitter and jitter alarms overlap.

3. The threshold recommendation method for adaptively reducing jitter alarm according to claim 1, characterized in that: The adjustable model structure of the xgboost model in step 3 includes: weighing the evaluation index according to the noise reduction rate and the accuracy rate, weighing the category weight according to the time taken from the alarm being turned on to being turned off, and weighing the sample weight according to the severity level of the alarm.

4. The threshold recommendation method for adaptively reducing jitter alarm according to claim 3 is characterized in that: The evaluation index based on the trade-off between noise reduction rate and accuracy is F β The noise reduction rate and accuracy are weighed and the segmentation score is calculated. The calculation formula is as follows: in, F β It is the calculation result of the custom F1 calculation formula; β is the penalty term. When β=1, it is the standard F1 calculation formula. When β>1, the accuracy is penalized. When β<1, the denoising rate is penalized. P is the noise reduction rate, and R is the accuracy rate.

5. The threshold recommendation method for adaptively reducing jitter alarm according to claim 1, characterized in that: The xgboost model in step 3 can be replaced by a catboost model or a lightgbm model.

6. The threshold recommendation method for adaptively reducing jitter alarm according to claim 1, characterized in that: The operation and maintenance personnel experience in step 3 is to provide relevant interactive pages for the front end based on the classification and summary of historical information.

7. The threshold recommendation method for adaptively reducing jitter alarm according to claim 1, characterized in that: The incremental learning refers to injecting new data into a trained model within a set period, so that the model can relearn and update the data into the database.

Images