Alarm noise reduction method based on AlertManager

By using time series aggregation and dynamic threshold adjustment methods in AlertManager, the redundant alarm problem in traditional monitoring alarm systems is solved, and more accurate and efficient alarm information processing is achieved.

CN119088650APending Publication Date: 2024-12-06INSPUR TIANYUAN COMM INFORMATION SYST CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411074827.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-07
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Traditional monitoring and alarm systems generate a large amount of redundant alarm information, resulting in high noise, affecting their targetedness and accuracy.

Method used

Using the AlertManager-based method, through time series alarm aggregation and dynamic threshold adjustment of machine learning, the FP-growth algorithm and soft-margin SVM model are used to cluster and filter redundant alarm information to adjust the alarm threshold in real time.

Benefits of technology

It effectively reduces noise in the alarm system, improves the pertinence and accuracy of alarm information, and reduces the transmission of redundant alarms.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119088650A_ABST
    Figure CN119088650A_ABST
Patent Text Reader

Abstract

The invention provides a method for realizing alarm noise reduction based on AlertManager, belongs to the technical field of computers, and effectively reduces noise in an alarm system and improves pertinence and accuracy of alarm information by introducing an alarm aggregation method based on a time sequence and a dynamic threshold adjustment method based on machine learning. According to the alarm aggregation method based on the time sequence, clustering is carried out through a relevance algorithm, sending of alarm information is effectively reduced, noise in an alarm system is reduced, an alarm threshold value is dynamically adjusted in real time through a dynamic threshold value adjusting method based on machine learning, and pertinence and accuracy of the alarm information are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method for realizing alarm noise reduction based on AlertManager. Background Art

[0002] With the rapid development of Internet technology, various complex systems and large-scale computer clusters are gradually increasing. These systems may often have various failures and anomalies. In order to solve these problems, real-time monitoring and alarm mechanisms have become necessary means. However, traditional monitoring and alarm systems often generate a large amount of redundant alarm information, which brings great trouble to relevant personnel. Therefore, how to reduce the noise in the alarm system and make the alarm information it generates more targeted and accurate is a problem that needs to be solved urgently. Summary of the invention

[0003] In order to solve the above technical problems, the present invention provides a method for realizing alarm noise reduction based on AlertManager, which reduces the noise in the alarm system, improves the pertinence and accuracy of the alarm information, and solves the problem of high noise in the alarm system caused by redundant alarm information.

[0004] The technical solution of the present invention is:

[0005] A method for realizing alarm noise reduction based on AlertManager includes: first, an alarm aggregation method based on time series, which collects the alarm information of AlertManager and clusters the related alarm information by using the bitmap-based FP-growth algorithm; second, a dynamic threshold adjustment method based on machine learning, which constructs a threshold model by using the soft-margin SVM algorithm through machine learning training on historical alarm data, and constructs a dynamic threshold model by training the continuously added new alarm information. The model can adjust the alarm threshold in real time according to the system status.

[0006] The specific steps include:

[0007] S1 obtains relevant alarm information from the AlertManager database;

[0008] S2 divides the alarm information into time series according to the time window size set by the system administrator;

[0009] S3 traverses each time window and performs bitmap FP-growth correlation analysis on the alarm information in each time window;

[0010] S4 calculates the cosine similarity between the warning information;

[0011] S5 aggregates the alarm information according to the similarity and generates a representative alarm;

[0012] S6 The alarm information enters the dynamic threshold model, and determines whether to send the alarm information to AlertManager based on the relationship between the model judgment result and the threshold;

[0013] S7 pushes representative alarms to the AlertManager for further processing.

[0014] Furthermore,

[0015] The bitmap-based FP-growth correlation analysis in step S3 includes the following steps:

[0016] (1) Bitmap construction: First, the data set D needs to be sorted from small to large according to the size of the item set, and then each item set is traversed and its corresponding bit is marked;

[0017] (2) Frequent item set mining: After the bitmap is constructed, frequent item set mining is performed on the bitmap through bit operations and set operations;

[0018] Among them, the bitmap-based FP-growth optimization is to build a bitmap index for each item set when traversing the data set for the first time, which is used to quickly filter frequent item sets; and calculate the intersection and union of frequent item sets through the AND and OR operations of the bitmap.

[0019] The frequent item set mining in step (2) includes the following steps:

[0020] A1 selects the item sets whose occurrence times are greater than or equal to minsup from the bitmap as frequent 1-item sets;

[0021] A2 builds a conditional pattern base, scans each frequent 1-item set vertically in the bitmap, obtains their locations, and eliminates item sets that do not meet the minimum support requirements;

[0022] A3 For each frequent 1-item set, construct its corresponding conditional FP tree using the conditional pattern base;

[0023] A4 For each frequent 1-item set's conditional FP tree, recursively mine its frequent item sets, and perform pruning and support calculation based on the conditional FP tree;

[0024] A5 compresses and stores the original bitmap to obtain a new array. Each element of the array is the difference from the average value of the array. If the difference is greater than the difference, it is a number 1-9, and if the difference is less than the difference, it is a lowercase letter aj. By reducing the length of the stored bitmap by the difference from the average value, the storage space is reduced, thereby optimizing the space size and speeding up the prediction speed.

[0025] A6 combines all the mined frequent item sets to obtain the final frequent item set list L.

[0026] The alarm information in step S1 includes a fault or abnormal event in the system, which needs to be detected and processed in time. The means for obtaining the alarm information includes monitoring software, log collector, sensor, and equipment indicators;

[0027] The time window in step S2 is a subset of the alarm data acquired within a certain time period;

[0028] The alarm information in step S3 includes the alarm source, alarm type, alarm level, alarm frequency, and alarm duration.

[0029] The method for constructing the dynamic threshold in step S6 comprises the following steps:

[0030] B1 collects historical alarm data;

[0031] B2 pre-processes historical alarm data and removes abnormal data;

[0032] B3 uses an improved version of the support vector machine model to train the preprocessed data and build a dynamic threshold model.

[0033] In step S6, when the alarm information is sent, the data transmission method includes the following steps:

[0034] 1) Segmentation: Split the original data into several blocks.

[0035] 2) Dictionary construction: During the compression process, the LZMA algorithm uses a dictionary to store strings that have appeared. The dictionary is a sliding window of fixed size or a dynamic dictionary of adaptive size;

[0036] 3) Matching search: For each block, the LZMA algorithm uses matching search to find repeated strings. It searches for the longest matching string in the dictionary and records the matching position and length information.

[0037] 4) Symbol encoding: Use range encoding technology to further reduce storage space and encode matching position and length information;

[0038] 5) Data compression: Use predictive compression algorithm to achieve high compression rate;

[0039] 6) Decompression: After receiving the compressed data block, the receiver decompresses each block;

[0040] 7) Data transmission: The decompressed data is transmitted to the application layer for further processing.

[0041] The prediction compression algorithm in step 5) includes the following steps:

[0042] 5.1) Initialize the prediction model: Initialize the prediction model based on the compressed data, including selecting appropriate prediction algorithms and parameters, and setting the initial state;

[0043] 5.2) Predict the next symbol or string: Use the neural network prediction algorithm model to predict the next symbol or string based on the compressed data;

[0044] 5.3) Calculate the prediction error: Use the mean square error method to compare the predicted symbol with the actual symbol and calculate the prediction error;

[0045] 5.4) Symbol encoding: Use Huffman coding method to encode the prediction error and convert it into compressed data;

[0046] 5.5) Output the encoding result: Output the encoded result to the compressed data so that the data can be correctly restored when decompressed;

[0047] 5.6) Update the prediction model: Update the parameters of the prediction model according to the actual sign and prediction error, and adjust the prediction model through the learning algorithm.

[0048] Representative methods for pushing the alarm to the AlertManager in step S7 include enterprise WeChat, API, and Webhook.

[0049] The dynamic threshold adjustment method comprises the following steps:

[0050] Data collection: Collect data for training and adjusting thresholds;

[0051] Feature extraction: extracting features from data for decision making;

[0052] Initial threshold setting: Set an initial threshold, which can be selected based on previous experimental results;

[0053] Real-time detection: Real-time detection and analysis of new data, i.e. real-time data of AlertManager;

[0054] User feedback module: You can manually mark misjudged emails, and the system adjusts the threshold based on user feedback;

[0055] Threshold adjustment: According to the characteristics and changes of the implementation data, the threshold is dynamically adjusted through the soft margin SVM algorithm and updated in real time;

[0056] Multi-model integration: Combine the random forest algorithm and the boosting algorithm, and determine the final result through the output of the base classifier.

[0057] The beneficial effects of the present invention are

[0058] By introducing the alarm aggregation method based on time series and the dynamic threshold adjustment method based on machine learning, the noise in the alarm system is effectively reduced, and the pertinence and accuracy of the alarm information are improved. The alarm aggregation method based on time series clusters through the correlation algorithm, effectively reducing the sending of alarm information, thereby reducing the noise in the alarm system, and the dynamic threshold adjustment method based on machine learning dynamically adjusts the alarm threshold in real time, thereby improving the pertinence and accuracy of the alarm information. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a schematic diagram of the AlertManager alarm method of the present invention;

[0060] Figure 2 This is a diagram of the alarm aggregation method based on time series;

[0061] Figure 3 It is a dynamic threshold model diagram based on soft margin SVM;

[0062] Figure 4 This is a schematic diagram of the cloud SVM architecture. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0064] The present invention provides a method for implementing alarm noise reduction based on AlertManager, comprising the following steps:

[0065] S1 obtains relevant alarm information from the AlertManager database;

[0066] S2 divides the alarm information into time series according to the time window size set by the system administrator;

[0067] S3 traverses each time window and performs bitmap FP-growth correlation analysis on the alarm information in each time window;

[0068] S4 calculates the cosine similarity between the warning information; (Cosine similarity is a common method for calculating text similarity, and is also applicable to other types of vector data. It can be used to measure the angle between two vectors, and its value range is between [-1,1]. When the angle between two vectors is 0, the cosine similarity is 1, indicating that the two vectors are exactly the same; when the angle between two vectors is 180 degrees, the cosine similarity is -1, indicating that the two vectors are completely opposite; when the angle between two vectors is 90 degrees, the cosine similarity is 0, indicating that there is no correlation between the two vectors.)

[0069] S5 aggregates the alarm information according to the similarity and generates a representative alarm;

[0070] S6 The alarm information enters the dynamic threshold model, and determines whether to send the alarm information to AlertManager based on the relationship between the model judgment result and the threshold;

[0071] S7 pushes representative alarms to the AlertManager for further processing.

[0072] The bitmap-based FP-growth algorithm method in step S3 comprises the following steps:

[0073] (1) Bitmap construction: First, the data set D needs to be sorted from small to large according to the size of the item set, and then each item set is traversed and its corresponding bit is marked;

[0074] (2) Frequent item set mining: After the bitmap is constructed, efficient frequent item set mining can be performed on the bitmap through bit operations and fast set operations.

[0075] Among them, the bitmap-based FP-growth optimization is to build a bitmap index for each item set when traversing the data set for the first time, which is used to quickly filter frequent item sets. The intersection and union of frequent item sets are efficiently calculated through the AND and OR operations of the bitmap.

[0076] The advantage of bitmap-based FP-growth is that by using bitmap AND and OR operations, bit-level logical operations can be performed efficiently, reducing computing time and resource overhead and improving the algorithm's operating efficiency.

[0077] The frequent item set mining method in step (2) comprises the following steps:

[0078] A1. Select the item sets whose occurrence times are greater than or equal to minsup from the bitmap as frequent 1-item sets;

[0079] A2. Build a conditional pattern base, scan each frequent 1-item set vertically in the bitmap, obtain their locations, and eliminate the item sets that do not meet the minimum support requirements;

[0080] A3. For each frequent 1-item set, construct its corresponding conditional FP tree using the conditional pattern base;

[0081] A4. For each frequent 1-item set’s conditional FP tree, recursively mine its frequent itemsets, and perform pruning and support calculation based on the conditional FP tree;

[0082] A5. After compressing and storing the original bitmap, a new array is obtained. Each element of the array is the difference of the average value of the array. If it is greater than the difference, it is a number 1-9, and if it is less than the difference, it is a lowercase letter aj. By reducing the length of the stored bitmap by the difference from the average value, the storage space is reduced, thereby optimizing the space size and speeding up the prediction speed.

[0083] Assuming there is an original bitmap array B = [b1, b2, ..., bn], then the original bitmap can be used to calculate and obtain a new array D = [d1, d2, ..., dn], where each di represents the difference from the average value of this array. This method can be more effectively compressed and stored when there are many repeated bits, improving the efficiency and scalability of the algorithm.

[0084] A6. Merge all the mined frequent item sets to obtain the final frequent item set list L.

[0085] The alarm information in step S1 includes possible faults or abnormal events in the system, which need to be detected and processed in a timely manner. The means for obtaining the alarm information include monitoring software, log collectors, sensors, and equipment indicators.

[0086] The time window in step S2 is a subset of the alarm data acquired within a certain time period.

[0087] The alarm information in step S3 includes the alarm source, alarm type, alarm level, alarm frequency, and alarm duration.

[0088] The method for constructing the dynamic threshold in step S6 comprises the following steps:

[0089] B1. Collect historical alarm data, including alarm source, alarm type, alarm time, alarm level, whether the alarm has been processed, historical alarm frequency, etc.

[0090] B2. Pre-process historical alarm data and remove abnormal data;

[0091] B3. Use the improved support vector machine model to train the preprocessed data and build a dynamic threshold model.

[0092] In step S6, when the alarm information is sent, the data transmission method includes the following steps:

[0093] 1. Segmentation: Split the original data into multiple blocks.

[0094] 2. Dictionary construction: During the compression process, the LZMA algorithm uses a dictionary to store strings that have appeared. The dictionary can be a sliding window of fixed size or a dynamic dictionary of adaptive size;

[0095] 3. Matching search: For each block, the LZMA algorithm uses matching search to find repeated strings. It searches for the longest matching string in the dictionary and records the matching position and length information.

[0096] 4. Symbol encoding: Use range encoding technology to further reduce storage space and encode the matching position and length information;

[0097] 5. Data compression: Use predictive compression algorithm to achieve high compression rate;

[0098] 6. Decompression: After receiving the compressed data blocks, the receiver decompresses each block;

[0099] 7. Data transfer: The decompressed data is transferred to the application layer for further processing.

[0100] The prediction compression algorithm in step (1) comprises the following steps:

[0101] 1. Initialize the prediction model: Initialize the prediction model based on the compressed data, including selecting appropriate prediction algorithms and parameters, and setting the initial state;

[0102] 2. Predict the next symbol or string: Use the neural network prediction algorithm model to predict the next symbol or string based on the compressed data;

[0103] 3. Calculate the prediction error: Use the mean square error method to compare the predicted symbol with the actual symbol and calculate the prediction error;

[0104] 4. Symbol encoding: Use Huffman coding method to encode the prediction error and convert it into compressed data;

[0105] 5. Output the encoding result: Output the encoded result to the compressed data so that the data can be correctly restored when decompressed;

[0106] 6. Update the prediction model: Update the parameters of the prediction model according to the actual sign and prediction error. The prediction model can be further adjusted through the learning algorithm to improve the accuracy of the next prediction.

[0107] It achieves efficient data compression and decompression, and can effectively solve problems such as low compression rate, high computing resources and time costs, and difficulty in decompressing compressed data.

[0108] Representative methods for pushing the alarm to A1ertManager in step S7 include enterprise WeChat, API, and Webhook.

[0109] The following are the existing algorithms:

[0110] Here we consider using the SVM (Support Vector Machine) algorithm to build a classification model. Support vector machine is a binary classification model whose goal is to find a decision boundary (also called a hyperplane) that maximizes the interval between two categories. In SVM, data usually consists of feature vectors and corresponding labels.

[0111] Assume there is a training data set D = {(x1, y1), (x2, y2)....(x n ,y n )}, where x i ∈R d represents the input vector, y i ∈{-1, 1} represents the corresponding label. The goal of SVM is to find a hyperplane ωx+b=0 that maximizes the interval between the two categories, that is, maximizes the margin. The hyperplane ωx+b=0 can be expressed as ω T x+b=0, where ω is the normal vector and b is the intercept.

[0112] The decision function of SVM can be expressed as: f(x)=sign(ω T x+b).

[0113] According to the goal of SVM: maximize the interval, the following constraints need to be met:

[0114]

[0115] These constraints can be simplified to:

[0116] y i (ω T x i +b)≥1

[0117] At the same time, the objective function of SVM can be expressed as:

[0118] , where ||ω|| represents the second norm of ω.

[0119]

[0120] sty i (ω T x i +b)≥1

[0121] Rewriting the objective function into a convex quadratic programming form, we can obtain:

[0122]

[0123] st y i (ω T x i +b)≥1

[0124] By solving the optimization problem, the best hyperplane ωx+b=0 can be obtained, thereby obtaining the decision function f(x) of SVM, which is used to classify new input data.

[0125] In traditional SVM, it is assumed that data is linearly separable, and the two types of data are completely separated by a hyperplane under the premise of maximizing the interval. However, in reality, there is often a certain amount of noise and outliers, which will cause the traditional SVM to overfit the training data during training and thus fail to generalize to unseen data. To solve this problem, the soft margin method can be used to allow some data points to fall into the margin content, thereby reducing the sensitivity of the model. Specifically, the original constraints can be:

[0126] y i (ω T x i +b)≥1

[0127] Modified to:

[0128] y i (ω T x i +b)≥1-ξi

[0129] where ξ i ≥0 is a slack variable, indicating that some sample points can be within the margin or misclassified. Obviously, when ξ i =0, the sample point is outside the margin and is not misclassified; when 0<ξ i When ξ is less than 1, the sample point is located inside the margin and is not correctly classified. i When >1, the sample point is misclassified.

[0130] In order to maximize the margin, we need to minimize all ξ i sum:

[0131]

[0132] Where C is the regularization parameter, which controls the trade-off between margin and misclassified samples. When C is small, it means that more attention is paid to the size of margin, and when C is large, more attention is paid to the accuracy of classification. At the same time, the constraints need to be met:

[0133]

[0134] Convert the above problem into a convex quadratic programming problem:

[0135]

[0136] sty i (ω T x i +b)≥1-ξ i ,

[0137] ξ i ≥0, i=1, ..., n

[0138] By solving this equation, we can obtain the optimal hyperplane and the corresponding slack variable ξ i , thus obtaining the decision function f(x) of SVM.

[0139] Now we design a new loss function, taking into account the size of the interval and the penalty for misclassified points, and improve the loss function of the softmargin SVM. The final loss function can be expressed as:

[0140]

[0141] sty i (w T x i +b)≥1-ξ i ,

[0142] ξ i ≥0, i=1, 2, ..., n

[0143] A trade-off factor a is introduced to control the size of the interval, and another weight factor b is introduced to control the penalty of misclassified points. A larger a value will tend to select a larger interval but may introduce some misclassified support vectors. A smaller a value will focus more on reducing misclassification but may lead to a smaller interval. According to the values ​​of a and b parameters, the interval size and misclassification can be balanced, which will achieve better classification results than soft margin SVM.

[0144] The dynamic threshold adjustment method is a method based on the characteristics and changes of real-time data, which achieves more accurate classification and discrimination by dynamically adjusting the threshold. Specifically, the dynamic threshold adjustment method includes the following steps: Data collection: Collect enough data for training and adjusting the threshold. These data are training data (historical data)

[0145] Feature extraction: extracting features from data for decision making

[0146] Initial threshold setting: To start the tuning process, an initial threshold needs to be set. This can be done by setting a default value based on experience. This can be selected based on previous experimental results.

[0147] Real-time detection: Real-time detection and analysis of new data, namely the real-time data user feedback module of AlertManager: users can manually mark misjudged emails, and the system adjusts the threshold based on user feedback to improve accuracy.

[0148] Threshold adjustment: According to the characteristics and changes of the implementation data, the threshold is dynamically adjusted through the soft margin SVM algorithm and updated in real time.

[0149] Multi-model integration: Combine the random forest algorithm and the boosting algorithm to determine the final result through the output results of multiple base classifiers.

[0150] CloudSVM is a MapReduce-based SVM training algorithm that runs in parallel on multiple commodity computers with Hadoop. Figure 4 As shown, the training set of the algorithm is divided into subsets, and each subset is evaluated separately to obtain the α value (i.e., support vector). In the map phase of the MapReduce job, the subset of the training set is combined with the global support vector. In the reduce step, the merged subset of training data is evaluated. The new support vector obtained is combined with the global support vector in the reduce step. CloudSVM using MapReduce algorithm can be explained as follows. First, each computer in the cloud computing system reads the global support vector, and then the global sv is merged with the subset of local training data and classified by SVM. Finally, all the calculated svv in the cloud computers are merged. Therefore, the algorithm uses the new sv to save the global sv. The algorithm of CloudSVM consists of the following 6 steps

[0151] 1. As initialization, the global support vector is set to t = 0,

[0152] 2. t = t + 1;

[0153] 3. For any l, l is between 1 and L, read the global SV and merge all SVs in the training set subset; 4. Use the merged new dataset to train the SVM algorithm;

[0154] 5. Find the support vector;

[0155] 6. After all computers in the cloud system complete the training phase, merge all calculated SVs and save the results to the global SV;

[0156] 7. If ht=ht-1 stop, otherwise go to step 2.

[0157] The pseudo codes of the mapping and simplification functions of the CloudSVM algorithm are given in Algorithm 1 and Algorithm 2 respectively:

[0158] Algorithm 1 is the mapping function of the CloudSVM algorithm

[0159]

[0160] Algorithm 2 reduces the function of CloudSVM algorithm

[0161]

[0162] To train the SVM classifier function, LibSVM with different kernels was used. The appropriate values ​​of parameters C and γ were found by cross-validation. All systems were implemented using Hadoop and the streaming Python package mrjob library. Convergence of Cloud Support Vector Machines

[0163] Let S denote a subset of the training set D, F(S) be the optimal objective function on the dataset S, and h* be the global optimal hypothesis with the smallest empirical risk Remp(h). The algorithm starts with SVOGlobal=0 and generates a non-increasing sequence of positive vectors SVtGlobal, where SVtGlobal is the vector of support vectors at the tth iteration. The model is tested using the hinge loss, a model trained using the CloudSVM algorithm. Hinge loss works well in SVM as a classifier because the more margins you violate, the higher the penalty

[20] . The hinge loss function is as follows:

[0164] l(f(x),y)=max{0,1-yf(x)}

[0165] The empirical risk can be calculated using an approximation:

[0166]

[0167] According to the empirical risk minimization principle, the learning algorithm should choose a hypothesis h that minimizes the empirical risk:

[0168]

[0169] There is a hypothesis in each cloud node. Let X be a subset of the training data at cloud node i, where X∈Rmxn, SVt Global is the vector of support vectors at iteration tt, and ht,i is the hypothesis at node i at iteration t. Then the optimization problem in Equation 3 is:

[0170]

[0171] Where Q12 and Q21 are the kernel matrices

[0172]

[0173] α1 and α2 are the solutions estimated by node i using the dataset X and SVGlobal. Due to Mercer's theorem, the kernel matrix Q is a symmetric positive definite function on a square. Then the sub-matrices Q12 and Q21 must be equal. The Q11 and Q22 matrices can be defined such that

[0174]

[0175] Q 22 ={K i,j (S Global , S Global )|i=1....,m,j=1....,n}

[0176] At iteration t, the algorithm reaches its stopping point when the empirical risk of the hypothesis is the same as in the previous iteration. That is:

[0177] R emp (h t )=R emp (h t-1 ) (5)

[0178] Lemma: The accuracy of the decision function of the CloudSVM classifier at iteration t is always greater than or equal to the maximum accuracy of the decision function of the SVM classifier at iteration t-1. This is also true:

[0179]

[0180] Proof: Iterative Cloud support vector machine converges monotonically to the optimal classifier.

[0181]

[0182] Where n is the split size of the data set (or cloud node size). Then, the training set for the SVM algorithm at node i is.

[0183]

[0184] Adding more samples does not reduce the optimal value. The accuracy of the subproblem at each node improves monotonically at each step.

[0185] The above description is only a preferred embodiment of the present invention, which is only used to illustrate the technical solution of the present invention, and is not used to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for implementing alarm noise reduction based on AlertManager, characterized in that: The alarm aggregation method based on time series collects the alarm information of AlertManager and clusters the related alarm information based on the bitmap FP-growth algorithm; The dynamic threshold adjustment method based on machine learning conducts machine learning training on historical alarm data, builds a threshold model through the soft-margin SVM algorithm, and builds a dynamic threshold model by training the continuously added new alarm information. The dynamic threshold model adjusts the alarm threshold in real time according to the system status.

2. The method according to claim 1, characterized in that include Follow these steps: S1 obtains relevant alarm information from the AlertManager database; S2 divides the alarm information into time series according to the time window size set by the system administrator; S3 traverses each time window and performs bitmap FP-growth correlation analysis on the alarm information in each time window; S4 calculates the cosine similarity between the warning information; S5 aggregates the alarm information according to the similarity and generates a representative alarm; S6 The alarm information enters the dynamic threshold model, and determines whether to send the alarm information to the AlertManager based on the relationship between the model judgment result and the threshold; S7 pushes representative alarms to the AlertManager for further processing.

3. The method according to claim 2, characterized in that The bitmap-based FP-growth correlation analysis in step S3 includes the following steps: (1) Bitmap construction: First, the data set D needs to be sorted from small to large according to the size of the item set, and then each item set is traversed and its corresponding bit is marked; (2) Frequent item set mining: After the bitmap is constructed, frequent item set mining is performed on the bitmap through bit operations and set operations; Among them, the bitmap-based FP-growth optimization is to build a bitmap index for each item set when traversing the data set for the first time, which is used to quickly filter frequent item sets; and calculate the intersection and union of frequent item sets through the AND and OR operations of the bitmap.

4. The method according to claim 3, characterized in that The frequent item set mining in step (2) includes the following steps: A1 selects the item sets whose occurrence times are greater than or equal to minsup from the bitmap as frequent 1-item sets; A2 builds a conditional pattern base, scans each frequent 1-item set vertically in the bitmap, obtains their locations, and eliminates item sets that do not meet the minimum support requirements; A3 For each frequent 1-item set, use the conditional pattern base to construct its corresponding conditional FP tree; A4 For each frequent 1-item set's conditional FP tree, recursively mine its frequent item sets, and perform pruning and support calculation based on the conditional FP tree; A5 compresses and stores the original bitmap to obtain a new array. Each element of the array is the difference from the average value of the array. If the difference is greater than the difference, it is a number 1-9, and if the difference is less than the difference, it is a lowercase letter aj. By reducing the length of the stored bitmap by the difference from the average value, the storage space is reduced, thereby optimizing the space size and speeding up the prediction speed. A6 combines all the mined frequent item sets to obtain the final frequent item set list L.

5. The method according to claim 2, characterized in that: The alarm information in step S1 includes a fault or abnormal event in the system, which needs to be detected and processed in time. The means for obtaining the alarm information includes monitoring software, log collector, sensor, and equipment indicators; The time window in step S2 is a subset of the alarm data acquired within a certain time period; The alarm information in step S3 includes the alarm source, alarm type, alarm level, alarm frequency, and alarm duration.

6. The method according to claim 2, characterized in that The method for constructing the dynamic threshold in step S6 comprises the following steps: B1 collects historical alarm data; B2 pre-processes historical alarm data and removes abnormal data; B3 uses an improved version of the support vector machine model to train the preprocessed data and build a dynamic threshold model.

7. The method according to claim 2, characterized in that In step S6, when the alarm information is sent, the data transmission method includes the following steps: 1) Segmentation: Split the original data into several blocks. 2) Dictionary construction: During the compression process, the LZMA algorithm uses a dictionary to store strings that have appeared. The dictionary is a sliding window of fixed size or a dynamic dictionary of adaptive size; 3) Matching search: For each block, the LZMA algorithm uses matching search to find repeated strings. It searches for the longest matching string in the dictionary and records the matching position and length information. 4) Symbol encoding: Use range encoding technology to further reduce storage space and encode matching position and length information; 5) Data compression: Use predictive compression algorithm to achieve high compression rate; 6) Decompression: After receiving the compressed data block, the receiver decompresses each block; 7) Data transmission: The decompressed data is transmitted to the application layer for further processing.

8. The method according to claim 7, characterized in that The prediction compression algorithm in step 5) includes the following steps: 5.1) Initialize the prediction model: Initialize the prediction model based on the compressed data, including selecting appropriate prediction algorithms and parameters, and setting the initial state; 5.2) Predict the next symbol or string: Use the neural network prediction algorithm model to predict the next symbol or string based on the compressed data; 5.3) Calculate the prediction error: Use the mean square error method to compare the predicted symbol with the actual symbol and calculate the prediction error; 5.4) Symbol encoding: Use Huffman coding method to encode the prediction error and convert it into compressed data; 5.5) Output the encoding result: Output the encoded result to the compressed data so that the data can be correctly restored when decompressed; 5.6) Update the prediction model: Update the parameters of the prediction model according to the actual sign and prediction error, and adjust the prediction model through the learning algorithm.

9. The method according to claim 1, characterized in that: Representative methods for pushing the alarm to the AlertManager in step S7 include enterprise WeChat, API, and Webhook.

10. The method according to claim 1, characterized in that The dynamic threshold adjustment method comprises the following steps: Data collection: Collect data for training and adjusting thresholds; Feature extraction: extracting features from data for decision making; Initial threshold setting: Set an initial threshold, which can be selected based on previous experimental results; Real-time detection: Real-time detection and analysis of new data, i.e. real-time data of AlertManager; User feedback module: You can manually mark misjudged emails, and the system adjusts the threshold based on user feedback; Threshold adjustment: According to the characteristics and changes of the implementation data, the threshold is dynamically adjusted through the soft margin SVM algorithm and updated in real time; Multi-model integration: Combine the random forest algorithm and the boosting algorithm, and determine the final result through the output of the base classifier.