Adaptive AIOps data drift detection method based on model key features

Abstract

The present invention provides an adaptive AIOps data drift detection method based on key features of a model, and belongs to the field of intelligent operation and maintenance technology. The method extracts important features from the original features, and respectively calculates the similarity of the feature distribution of the model's training data and test data on the important features, and configures a concept drift detector according to the similarity evaluation results; performs feature importance evaluation on the features used to build the model, and selects a third feature related to the model performance from the evaluation results; builds a fault prediction AIOps model, and sets a baseline model degradation index and a performance evaluation index; based on the baseline model degradation index and the performance evaluation index, monitors the number of changes in the third feature during model training and testing, and detects the concept drift of the recognition model through a concept drift detector and issues a drift alarm. The present invention can timely discover the degradation of the AIOps model by real-time monitoring of changes in data features and fluctuations in model accuracy.

Description

Technical Field

[0001] The present invention relates to the field of intelligent operation and maintenance technology, and in particular to an adaptive AIOps data drift detection method based on model key features. Background Art

[0002] In the field of AIOps (Artificial Intelligence for IT Operations), concept drift is an important concept, which usually refers to the changes in data over time. Many drift detectors have been proposed for classification problems. Some drift detection techniques monitor the changes in model accuracy over time and rely on labeled data, while other techniques only identify drift in dataset characteristics (such as data distribution). Although label-dependent drift detection techniques are reliable indicators of model performance degradation, they are costly to use in real-world scenarios because true labels are not necessarily immediately available. Therefore, it is more appropriate to use drift detectors based on data distribution as an alternative. However, previous studies have shown that these types of detectors are very sensitive to small changes in the data that do not necessarily affect the accuracy of machine learning models.

[0003] In terms of evaluation, concept drift detectors are usually evaluated in two ways: evaluating drift detection accuracy and evaluating the performance of machine learning models over time. The former evaluation technique measures the ability of the drift detector to correctly label drift and non-drift, and is usually adopted when experimenting with synthetic data where fixed drift occurs. However, there is also a technique that uses a Z-test of the error rate to label the test batches as drift and non-drift respectively, which is more applicable to practical situations. In this evaluation technique, in order to understand the impact of the detector, the performance of the machine learning model retrained based on the drift detector is usually compared with the performance of the model that has never been retrained.

[0004] Concept drift can also cause degradation of AIOps models. Although AIOps models have a positive impact on software delivery acceleration and software system quality and security, the performance of these models does not generalize over time due to concept drift. AIOps models trained on current data do not generalize well to future data. Concept drift on operational data can negatively impact the performance of fault prediction AIOps models, causing their error rates to increase. Previous research has highlighted the importance of regularly updating AIOps models to mitigate the effects of concept drift, as well as the need to monitor AIOps models to prevent concept drift. The effects of retraining fault prediction models based on degradation indicator monitoring tools that require labels have been studied, but there is currently no research on unsupervised model degradation indicators.

[0005] Data monitoring plays an important role in model monitoring, and one technique is to monitor the number of features that deviate over time. Another popular data monitoring technique is to continuously check whether the distribution of the data changes over time. Although these monitoring techniques have been proposed, their ability to identify AIOps model degradation has not yet been evaluated. Summary of the invention

[0006] In view of this, an embodiment of the present invention provides an adaptive AIOps data drift detection method based on model key features to solve the problem that the existing AIOps system is not timely and accurate in monitoring model performance changes.

[0007] The technical solution adopted by the present invention is:

[0008] The present invention provides an adaptive AIOps data drift detection method based on model key features, comprising:

[0009] The important features are extracted from the original features through the feature importance selection method, and the first feature distribution and the second feature distribution of the model's training data and test data on the important features are calculated respectively. At the same time, the KS test is used to evaluate the similarity between the first feature distribution and the second feature distribution. The concept drift detector is configured according to the similarity evaluation results.

[0010] Preprocess the original data to obtain data sets for model training and model testing;

[0011] Perform feature importance evaluation on the features used to build the AIOps model, and select the third feature related to model performance from the feature importance evaluation results through a feature selection algorithm;

[0012] Build a fault prediction AIOps model through a random forest classifier and set benchmark model degradation indicators and performance evaluation indicators;

[0013] Based on the benchmark model degradation indicators and performance evaluation indicators, the number of third feature changes of the fault prediction AIOps model during training and testing using the dataset is monitored. At the same time, the concept drift of the recognition model is detected through the concept drift detector and a drift alarm is issued.

[0014] Furthermore, the important features are extracted from the original features by the feature importance selection method, and the first feature distribution and the second feature distribution of the training data and the test data of the model on the important features are respectively calculated, and the similarity between the first feature distribution and the second feature distribution is evaluated by the KS test, and the concept drift detector is configured according to the similarity evaluation result, including:

[0015] The average impurity reduction method is selected as the feature importance selection method, the original features are sorted, the average importance ranking of all original features is obtained, and the average value of the average importance ranking is calculated as the feature extraction threshold, and the original features with a ranking higher than the feature extraction threshold are taken as important features;

[0016] Calculate the first feature distribution of the model's training data on the important features and the second feature distribution of the test data on the important features;

[0017] A KS test is used to evaluate the similarity or difference between the first feature distribution and the second feature distribution, a drift alarm threshold is determined according to the evaluation result of the similarity or difference, and a concept drift detector is configured based on the drift alarm threshold.

[0018] Furthermore, the raw data is preprocessed to obtain data sets for model training and model testing, including:

[0019] Obtain the original data for model training and testing, and clean the original data to remove noise and outliers;

[0020] Apply standardization or normalization methods to the raw data after data cleaning to a unified scale, and convert the characteristic values ​​of the converted raw data into a distribution with a mean of 0 and a standard deviation of 1 to obtain normalized data. x std :

[0021] ,

[0022] in, x is the original eigenvalue, μ is the mean of the feature, σ is the standard deviation of the feature;

[0023] For normalized data x std Missing value processing is performed to obtain a data set for model training and model testing, wherein the data set includes a training set and a test set.

[0024] Furthermore, the feature importance evaluation is performed on the features used to construct the AIOps model, and a third feature related to the model performance is selected from the feature importance evaluation results by a feature selection algorithm, including:

[0025] The average information gain of the features used to build the AIOps model is calculated using a feature importance scoring method, and the feature importance is evaluated based on the average information gain of the features to obtain a feature importance evaluation result; the features used to build the AIOps model include time features and configuration features;

[0026] Based on the feature importance evaluation results, the third feature that affects the model performance is screened out from the features used to build the AIOps model through the feature selection algorithm.

[0027] Furthermore, the process of setting the performance evaluation index includes:

[0028] The fault prediction AIOps model is trained and tested based on the training set and test set in the dataset, and the concept drift detector is used to detect drift in the training and testing process of the fault prediction AIOps model;

[0029] The drift detection accuracy evaluation technology is used to evaluate the drift detection accuracy of the concept drift detector, and the drift detection metric is determined based on the evaluation results.

[0030] Retrain the fault prediction AIOps model based on the drift detection results, compare the performance difference between the retrained fault prediction AIOps model and the periodically updated fault prediction AIOps model, and calculate the evaluation label loss based on the performance difference;

[0031] The performance evaluation index is obtained based on the drift detection metric and the evaluation label loss.

[0032] Furthermore, the drift detection accuracy evaluation technology is used to evaluate the drift detection accuracy of the concept drift detector, and the balanced accuracy of the drift detection is determined according to the evaluation result, including:

[0033] The two-proportion Z test method is used to evaluate whether there is concept drift between the training set and the test set in the recognition data set based on the error rate. The specific evaluation calculation formula is:

[0034]

[0035] in, is the prediction error rate on the training set, is the prediction error rate on the test set, is the overall prediction error rate, is the length of the training set, and is the length of the test set;

[0036] If there is no statistical difference between the performance of the model on the training set and the performance on the test set, there is no concept drift. If there is a statistical difference between the performance of the model on the training set and the performance on the test set, there is concept drift, and the severity of the concept drift is calculated. The calculation formula for the severity of concept drift is:

[0037] ;

[0038] Based on the severity of concept drift, balanced accuracy, specificity, and sensitivity are used as metrics to evaluate the accuracy of drift detection;

[0039] Specificity The calculation formula is: ,

[0040] Sensitivity The calculation formula is: ;

[0041] Among them, TN represents the number of instances correctly predicted by the model as negative samples, FN represents the number of instances incorrectly predicted by the model as negative samples, TP represents the number of instances correctly predicted by the model as positive samples, and FP represents the number of instances incorrectly predicted by the model as positive samples;

[0042] The calculation formula of balanced accuracy BalancedAccuracy is as follows:

[0043] .

[0044] Furthermore, retraining the fault prediction AIOps model based on the drift detection result includes:

[0045] A sliding window-based model retraining technology is used. When data drift is detected in the model, the fault prediction AIOps model is retrained based on the sliding window using a training set that only includes the identified data drift.

[0046] In summary, the beneficial effects of the present invention are as follows:

[0047] The present invention provides an adaptive AIOps data drift detection method based on key features of the model. The method extracts important features from the original features through a feature importance selection method, and respectively calculates the first feature distribution and the second feature distribution of the model's training data and test data on the important features. At the same time, the KS test is used to evaluate the similarity between the first feature distribution and the second feature distribution. A concept drift detector is configured according to the similarity evaluation result. Combining advanced machine learning algorithms and data analysis techniques, the detector can accurately monitor changes in data distribution in the AIOps system and issue alarms in a timely manner. At the same time, the present application combines concept drift detection technology with data monitoring technology to continuously monitor changes in data distribution and changes in model performance over time. By real-time monitoring of changes in data features and fluctuations in model accuracy, the degradation of the AIOps model can be discovered in a timely manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the following is a brief introduction to the drawings required for use in the embodiment of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work, and these are all within the protection scope of the present invention.

[0049] Figure 1 It is a schematic diagram of the process flow of the adaptive AIOps data drift detection method based on key features of the model of the present invention;

[0050] Figure 2 This is a schematic diagram of the concept drift judgment principle between two batches of data of the present invention;

[0051] Figure 3 A schematic diagram showing the difference between the periodic retraining of the present invention and the retraining based on drift detection;

[0052] Figure 4 A schematic diagram of label cost-effective maintenance training of the fault prediction model of the present invention;

[0053] Figure 5 Schematic diagram of batch distribution of drifted and non-drifted data sets at different time intervals of the present invention;

[0054] Figure 6 It is a schematic diagram of the percentage of feature changes in each time period of the present invention and its corresponding batch label. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. If there is no conflict, the various features of the present invention and the embodiments can be combined with each other, all within the scope of protection of the present invention.

[0056] Example 1: Reference Figure 1 As shown, Figure 1 This is a flow chart of the adaptive AIOps data drift detection method based on model key features of the present invention. Figure 1 As shown, the method mainly includes the following steps:

[0057] The important features are extracted from the original features through the feature importance selection method, and the first feature distribution and the second feature distribution of the model's training data and test data on the important features are calculated respectively. At the same time, the KS test is used to evaluate the similarity between the first feature distribution and the second feature distribution. The concept drift detector is configured according to the similarity evaluation results.

[0058] Preprocess the original data to obtain data sets for model training and model testing;

[0059] Perform feature importance evaluation on the features used to build the AIOps model, and select the third feature related to model performance from the feature importance evaluation results through a feature selection algorithm;

[0060] Build a fault prediction AIOps model through a random forest classifier and set benchmark model degradation indicators and performance evaluation indicators;

[0061] Based on the benchmark model degradation indicators and performance evaluation indicators, the number of third feature changes of the fault prediction AIOps model during training and testing using the dataset is monitored. At the same time, the concept drift of the recognition model is detected through the concept drift detector and a drift alarm is issued.

[0062] Furthermore, in an embodiment of the present invention, important features are extracted from original features by a feature importance selection method, and first feature distribution and second feature distribution of training data and test data of the model on the important features are calculated respectively, and the similarity between the first feature distribution and the second feature distribution is evaluated by a KS test, and a concept drift detector is configured according to the similarity evaluation result, including:

[0063] The average impurity reduction method is selected as the feature importance selection method. The original features are sorted to obtain the average importance ranking of all original features. The average value of the average importance ranking is calculated as the feature extraction threshold, and the original features with a ranking higher than the feature extraction threshold are taken as important features.

[0064] Among them, Mean Decrease in Impurity (MDI) is a purity-based feature importance method that sorts features, extracts the average importance ranking of all features, takes the average as the feature extraction threshold, and considers those features with a ranking higher than the average as relevant features, while the rest are discarded. Compared with the method of using a fixed percentage of important features, using average importance can improve the generalization ability of the detector.

[0065] Calculate the first feature distribution of the model's training data on important features and the second feature distribution of the test data on important features. Specifically, after filtering out unimportant features, it is necessary to estimate the data distribution. In this step, it is necessary to calculate the distribution of the training data and the test data on the most important features.

[0066] The KS test is used to evaluate the similarity or difference between the first feature distribution and the second feature distribution, the drift alarm threshold is determined according to the evaluation result of the similarity or difference, and the concept drift detector is configured based on the drift alarm threshold. Specifically, the embodiment of the present invention uses the KS statistical test (Kolmogorov-Smirnov test) to evaluate the similarity or difference between the distributions of the most important features in the training data and the test data. The null hypothesis of the test is that the two distributions are similar. When the p value is less than 0.05 (95% confidence interval), a drift alarm is issued (i.e., drift is detected).

[0067] Furthermore, in an embodiment of the present invention, the original data is preprocessed to obtain a data set for model training and model testing, including:

[0068] Obtain the original data for model training and testing, and clean the original data to remove noise and outliers;

[0069] Apply standardization or normalization methods to the raw data after data cleaning to a unified scale, and convert the characteristic values ​​of the converted raw data into a distribution with a mean of 0 and a standard deviation of 1 to obtain normalized data. x std :

[0070] ,

[0071] in, x is the original eigenvalue, μ is the mean of the feature, σ is the standard deviation of the feature;

[0072] For normalized data x std Missing value processing is performed to obtain a data set for model training and model testing, wherein the data set includes a training set and a test set.

[0073] Furthermore, in an embodiment of the present invention, feature importance evaluation is performed on the features used to construct the AIOps model, and a third feature related to model performance is selected from the feature importance evaluation results through a feature selection algorithm, including:

[0074] The average information gain of the features used to build the AIOps model is calculated using a feature importance scoring method, and the feature importance is evaluated based on the average information gain of the features to obtain a feature importance evaluation result; the features used to build the AIOps model include time features and configuration features.

[0075] Specifically, in order to build an AIOps model for pre-experimentation, 19 features were used, some of which had values ​​accumulated over time. When building an AIOps model for predicting failures, a total of 15 features were used, of which 6 were time features and 9 were configuration features. The present invention uses various feature selection techniques to evaluate the contribution of each feature to model performance. For example, feature importance scores (such as feature importance based on random forests) are used to perform evaluation by calculating the average information gain of each feature.

[0076] Based on the feature importance evaluation results, the third feature that affects the model performance is screened out from the features used to build the AIOps model through the feature selection algorithm.

[0077] Specifically, feature selection algorithms are mainly divided into three categories: filter, wrapper, and embedded.

[0078] Filtering algorithms: These algorithms do not rely on specific machine learning models, but instead use statistical tests or information gain methods to evaluate the importance of each feature. Common filtering algorithms include variance-based methods, chi-square tests, etc.

[0079] ‌Wrapper Algorithms‌: These algorithms directly use a specific machine learning model to evaluate the performance of a feature subset. Common wrapper algorithms include ‌Recursive Feature Elimination (RFE), ‌Feature Perturbation, etc.

[0080] ‌Embedded Algorithms‌: This type of algorithm embeds the feature selection process into the model training process. Common embedded algorithms include regularization-based methods, such as ‌LASSO regression. When screening features, you can choose according to actual needs.

[0081] In an embodiment of the present invention, a feature selection algorithm, such as recursive feature elimination (RFE) or a linear model with L1 regularization, is used to further screen features. The RFE algorithm optimizes model performance by gradually removing the least important features. L1 regularization automatically compresses the coefficients of certain features to zero by imposing a penalty on feature weights, thereby performing feature selection.

[0082] In the embodiment of the present invention, a random forest classifier is used to build a fault prediction AIOps model. First, a tree-based classifier is selected because this type of classifier has achieved good results in feature importance ranking extraction. Then, among the tree-based classifiers, we specifically selected random forest because it is a commonly used tree-based classifier in fault prediction AIOps solutions.

[0083] Furthermore, the present invention determines the baseline model degradation index, and identifies the degradation of the model through the model degradation index. First, the degradation of the model performance is captured by understanding the reflection of two data monitoring techniques, namely, monitoring the number of third features that change and monitoring the change of data distribution. When monitoring the number of features that change, the KS statistical test is used to calculate the number of features that are different from training and testing, so as to further evaluate how KS captures drift.

[0084] Furthermore, in the embodiment of the present invention, the process of setting the performance evaluation index includes:

[0085] The fault prediction AIOps model is trained and tested based on the training set and test set in the dataset, and the concept drift detector is used to detect drift in the training and testing process of the fault prediction AIOps model;

[0086] The drift detection accuracy evaluation technology is used to evaluate the drift detection accuracy of the concept drift detector, and the drift detection metric is determined based on the evaluation results.

[0087] Retrain the fault prediction AIOps model based on the drift detection results, compare the performance difference between the retrained fault prediction AIOps model and the periodically updated fault prediction AIOps model, and calculate the evaluation label loss based on the performance difference;

[0088] The performance evaluation index is obtained based on the drift detection metric and the evaluation label loss.

[0089] Specifically, in the embodiment of the present invention, two concept drift evaluation technologies are used, namely, drift detection accuracy evaluation and fault prediction model performance evaluation over time.

[0090] Furthermore, in an embodiment of the present invention, in the drift detection accuracy evaluation process, in order to evaluate the accuracy of the drift detector in correctly distinguishing drift from non-drift, it is first necessary to obtain the Groud Truth (reference true value) about which test batch is drifting and which is non-drifting.

[0091] In the process of extracting the benchmark true value, the present invention uses a drift detection accuracy evaluation technology to evaluate the drift detection accuracy of the concept drift detector, and determines the balance accuracy of the drift detection according to the evaluation result. The process specifically includes:

[0092] The model is trained on data corresponding to the previous time period and then evaluated on data from the current time period. It is assumed that the performance of the model on the training data (past data) is not statistically different from the performance on the test data (current data). If this assumption is not true, then concept drift exists.

[0093] Reference Figure 2 As shown, Figure 2 The principle of whether there is concept drift between datasets extracted from two different periods is described. The ML model is trained using data from the first period (P1) and tested on data from the second period (P2). The error rate at training time is calculated by performing 10-fold cross validation on data P1, and the error rate at test time is calculated by testing the model on data P2. We apply a two-proportion Z-test on these two error rates to evaluate whether there is concept drift between datasets from two different periods.

[0094] The two-proportion Z test method is used to evaluate whether there is concept drift between the training set and the test set in the recognition data set based on the error rate. The specific evaluation calculation formula is:

[0095]

[0096] in, is the prediction error rate on the training set, is the prediction error rate on the test set, is the overall prediction error rate, is the length of the training set, and is the length of the test set;

[0097] If there is no statistical difference between the performance of the model on the training set and the performance on the test set, there is no concept drift. If there is a statistical difference between the performance of the model on the training set and the performance on the test set, there is concept drift. Specifically, when the p value of the Z-test is lower than 0.05, the null hypothesis is rejected. At this time, the severity of concept drift is calculated. The calculation formula for the severity of concept drift is:

[0098] ;

[0099] Based on the severity of concept drift, balanced accuracy, specificity, and sensitivity are used as metrics to evaluate the accuracy of drift detection. In each dataset, the ratio between drift and non-drift is highly unbalanced. Three metrics are used here to take into account the imbalance between the two, namely balanced accuracy, specificity, and sensitivity. Specificity shows the ratio of the number of correctly identified drifts to the total number of drifts. Sensitivity shows the ratio of the number of correctly identified non-drifts to the total number of non-drifts. Balanced accuracy is the arithmetic mean between sensitivity and specificity, which shows the number of correctly classified drifts and non-drifts overall. Among them,

[0100] Specificity The calculation formula is: ,

[0101] Sensitivity The calculation formula is: ;

[0102] Among them, TN represents the number of instances correctly predicted by the model as negative samples, FN represents the number of instances incorrectly predicted by the model as negative samples, TP represents the number of instances correctly predicted by the model as positive samples, and FP represents the number of instances incorrectly predicted by the model as positive samples;

[0103] The balanced accuracy is obtained by comprehensively considering the specificity and sensitivity, which is used to evaluate the accuracy of drift detection. The balanced accuracy calculation formula is as follows:

[0104] .

[0105] Specifically, in the evaluation process based on model performance retention, the embodiment of the present invention analyzes the effects of not updating the model at all (static), updating the model periodically (the most commonly used technique for mitigating the impact of AIOps data concept drift), and updating the model based on drift detection technology. By comparing the model update based on drift detection technology with the static model update, it aims to verify whether using drift detection technology as a retraining indicator significantly improves the performance of the model compared to never updating the model. By comparing the model updated periodically with the model updated based on the drift detector, it is verified whether the drift detection technology can capture enough drift to achieve model performance comparable to periodic updates.

[0106] like Figure 3 As shown in Figure 1, periodic retraining updates the model after every fixed period of time, while drift detection-based retraining updates the model only when data drift is detected. A sliding window-based model retraining technique is used here, that is, the latest data is used during retraining.

[0107] The drift detection-based retraining method ensures that the model always reflects the latest data features while reducing unnecessary retraining (redundant iterations are inevitable in regular retraining), thereby improving the efficiency and performance of the model in the production environment.

[0108] Figure 3 The difference between periodic retraining and drift detection based retraining is described in , where parameters A, B, C represent data batch A, data batch B, and data batch C, respectively, and t represents the time period. The former updates the model once after each period, while the latter only updates the model when drift is detected. In the method of the embodiment of the present invention, the same sliding window based model retraining technique as the previous work is adopted, where the latest data is used for retraining.

[0109] Therefore, in an embodiment of the present invention, retraining the fault prediction AIOps model based on the drift detection result includes:

[0110] A sliding window-based model retraining technology is used. When data drift is detected in the model, the fault prediction AIOps model is retrained based on the sliding window using a training set that only includes the identified data drift.

[0111] Furthermore, in an embodiment of the present invention, in order to understand the benefits of drift detectors in reducing the number of model retraining and deployment times and the cost of obtaining high-quality labels, the present invention proposes a new evaluation strategy, as shown in FIG4. In this evaluation strategy, retraining is performed only when drift is indicated, and the training data set only includes data batch B in which drift has been identified, rather than including the latest data (batches A and B) in the training set as in other methods. If drift is not detected in batch A, the data distribution of batch A is similar to the data distribution in the training data that the model has previously been fitted to. Therefore, retraining batch A will result in inefficient training costs.

[0112] Furthermore, in the embodiment of the present invention, by continuously monitoring the real-time changes of the model performance and the data feature statistics, the statistical significance of the detected drift signals is calculated. If these drift signals exceed the drift alarm threshold, the drift alarm is triggered.

[0113] The embodiments of the present invention have the following technical features:

[0114] (1) Concept drift detector: The embodiment of the present invention develops a new type of concept drift detector, which combines advanced machine learning algorithms and data analysis techniques to accurately monitor changes in data distribution in the AIOps system and issue alarms in a timely manner.

[0115] (2) Combined application of data monitoring technology: The embodiment of the present invention combines concept drift detection technology with data monitoring technology to continuously monitor changes in data distribution and changes in model performance over time. By real-time monitoring of changes in data features and fluctuations in model accuracy, the degradation of the AIOps model can be discovered in a timely manner.

[0116] (3) Unsupervised model degradation index: The embodiment of the present invention determines the timing of retraining the model based on the unsupervised model degradation index of concept drift to ensure the stability of the AIOps system and maximize the performance and reliability of the system.

[0117] (4) Optimizing parameter settings: The embodiments of the present invention can be optimized according to different AIOps models and scenarios. By adjusting the parameters of model drift detection, the accuracy and stability of drift detection can be improved, thereby better adapting to different monitoring needs.

[0118] (5) Automated decision support: The embodiment of the present invention integrates a concept drift detector and a model degradation indicator to achieve real-time monitoring and analysis of the performance of the AIOps system. It can automatically identify potential model degradation situations and provide targeted suggestions and decisions to help the operation and maintenance team quickly respond to and solve problems.

[0119] (6) Evaluation and optimization of model degradation indicators: This paper not only proposes a new type of unsupervised model degradation indicator, but also fully evaluates and optimizes it. Through a large number of experimental verifications and data analysis, the accuracy and stability of the model degradation indicator are ensured, providing a reliable basis for the performance monitoring and management of the AIOps system.

[0120] (7) Optimization of the regular update strategy: Combining the concept drift detector and the model degradation indicator, this paper proposes an optimized regular update strategy that can dynamically adjust the model update frequency and timing according to actual conditions. Through a reasonable update strategy, the efficiency and availability of the AIOps system can be maximized, and maintenance costs and risks can be reduced.

[0121] (8) Scalability and flexibility: The method of the embodiment of the present invention can be customized and expanded according to different needs and scenarios. Whether it is a small-scale enterprise or a large organization, it can flexibly select and deploy a suitable monitoring solution according to its own needs and resource conditions to improve the management efficiency and operational stability of the AIOps system.

[0122] Embodiment 2: The primary purpose of the embodiment of the present invention is to solve the deficiencies in the existing AIOps system for monitoring model degradation and to provide a reliable method to accurately and timely monitor the performance degradation of the AIOps model. By real-time monitoring of concept drift and changes in model performance, the present invention aims to provide effective early warnings for the operation and maintenance team so that they can take timely measures to maintain the high availability and stability of the AIOps system, thereby minimizing the impact of system failures on business operations.

[0123] Specifically, when extracting the Groud Truth, we first extract the error rate of the classifier used, with a total of 10 random seeds, and then use the same intervals as previous work, that is, one dataset with one interval per day and another dataset with one interval per month. In most cases, all batches of the latter dataset drift. In order to understand the ability of the drift detector to identify drift without generating false positives, the period size is reduced from one month to one week. Because from our experiments, it can be seen that the data from one month to another varies greatly.

[0124] Reference Figure 5 As shown, Figure 5 The drifts identified based on error rates for two datasets (Disk dataset and Job dataset) are described, and their severity is reported as the relative difference in error rate. The data batch status is labeled as drift or non-drift. The drift label refers to the batches with more than half of the random seed number for the batches labeled as drift, while the other batches are labeled as non-drift. For the first dataset, due to class imbalance, the data batch corresponding to the first epoch contains only samples from the majority class, which makes the classifier unable to learn to distinguish between failures and non-failures. Therefore, it is impossible to analyze whether there is concept drift between the first and second epochs, so it is excluded from the figure.

[0125] Further, refer to Figure 6 As shown in Figure 6, the number of features that changed in each of two consecutive test batches is shown. Not all batches that are not marked as drifting have a smaller number of changed features, and sometimes batches marked as drifting contain fewer changed features. Therefore, model degradation is not related to the number of changed individual features.

[0126] Specifically, the embodiment of the present invention analyzes model-centric degradation indicators based on the number of monitored features and data distribution, and specifically evaluates drift detection accuracy (RQ2a) and performance retention over time (RQ2b).

[0127] Evaluation based on drift detection accuracy. The drift detection performance was compared with the degradation indicator (periodicity) for the updated fault prediction model and the degradation indicator (KS) that is not the center of the model. In further analysis, the random forest classifier based on the mean importance threshold considered about 7 out of 19 features as important. When it comes to the job dataset, about 9 out of 15 features are considered important. Therefore, when the classifier considers less than half of the features as important, the indicator achieves good results in correctly detecting non-drifts.

[0128] Evaluation based on performance retention. The effects of the concept drift detector in the present invention and KS are compared in the maintenance pipeline of the AIOps fault prediction model. The performance of the model without retraining (static) and the model with regular retraining (periodic) are used as the lower and upper baselines. Ten different random seeds are used to initialize the random forest classifier. The number of times the model needs to be retrained under each random seed is recorded, and then these numbers are averaged. This average reflects the frequency with which the model actually needs to be retrained during the entire experiment, relative to the maximum possible number of retraining. This approach ensures the robustness and repeatability of the experimental results, and provides a comprehensive evaluation of the model performance when dealing with concept drift. The experimental data is shown in Table 1 below:

[0129] Table 1. Comparison of model performance in handling concept drift

[0130]

[0131] Table 1 shows that the static model with a lower baseline performs the worst in terms of the ROCAUC (Receiver Operating Characteristic Area Under the Curve) curve indicator, and the periodic model with a higher baseline achieves the highest ROCAUC curve indicator, but requires continuous retraining (14 / 14). It can also be seen that the periodic and inventive detectors have high and similar performances in the ROCAUC indicator, but the inventive detector requires the least number of retraining times (11.9 times).

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An adaptive AIOps data drift detection method based on model key features, characterized in that: include: Extract important features from original features by feature importance selection method, calculate first feature distribution and second feature distribution of model training data and test data on important features respectively, use KS test to evaluate similarity between first feature distribution and second feature distribution, configure concept drift detector according to similarity evaluation result, including: select average impurity reduction method as feature importance selection method, sort original features to obtain average importance ranking of all original features, calculate average value of average importance ranking as feature extraction threshold, take original features with ranking higher than feature extraction threshold as important features; calculate first feature distribution of model training data on important features and second feature distribution of test data on important features; use KS test to evaluate similarity or difference between first feature distribution and second feature distribution, determine drift alarm threshold according to similarity or difference evaluation result, configure concept drift detector based on drift alarm threshold; Preprocess the original data to obtain data sets for model training and model testing; The features used to construct the AIOps model are evaluated for feature importance, and a third feature related to model performance is selected from the feature importance evaluation results by a feature selection algorithm, including: using a feature importance scoring method to calculate the average information gain of the features used to construct the AIOps model, and evaluating the feature importance according to the average information gain of the features to obtain a feature importance evaluation result; the features used to construct the AIOps model include time features and configuration features; based on the feature importance evaluation results, a third feature that affects model performance is selected from the features used to construct the AIOps model by a feature selection algorithm; Build a fault prediction AIOps model through a random forest classifier and set benchmark model degradation indicators and performance evaluation indicators; The process of setting the performance evaluation index includes: training and testing the fault prediction AIOps model based on the training set and the test set in the data set, and performing drift detection on the training and testing process of the fault prediction AIOps model using a concept drift detector; The drift detection accuracy evaluation technology is used to evaluate the drift detection accuracy of the concept drift detector, and the drift detection metric is determined based on the evaluation results, including: using the two-proportion Z test method to evaluate whether there is a concept drift between the training set and the test set in the recognition data set based on the error rate. The specific evaluation calculation formula is: in, is the prediction error rate on the training set, is the prediction error rate on the test set, is the overall prediction error rate, is the length of the training set, and is the length of the test set; If there is no statistical difference between the performance of the model on the training set and the performance on the test set, there is no concept drift. If there is a statistical difference between the performance of the model on the training set and the performance on the test set, there is concept drift, and the severity of the concept drift is calculated. The calculation formula for the severity of concept drift is: ； Based on the severity of concept drift, balanced accuracy, specificity, and sensitivity are used as metrics to evaluate the accuracy of drift detection; Specificity The calculation formula is: , Sensitivity The calculation formula is: ; Among them, TN represents the number of instances correctly predicted by the model as negative samples, FN represents the number of instances incorrectly predicted by the model as negative samples, TP represents the number of instances correctly predicted by the model as positive samples, and FP represents the number of instances incorrectly predicted by the model as positive samples; The calculation formula of balanced accuracy BalancedAccuracy is as follows: ； Retrain the fault prediction AIOps model based on the drift detection results, compare the performance difference between the retrained fault prediction AIOps model and the periodically updated fault prediction AIOps model, and calculate the evaluation label loss based on the performance difference; obtain the performance evaluation index based on the drift detection metric and the evaluation label loss; The retraining of the fault prediction AIOps model based on the drift detection result includes: adopting a model retraining technology based on a sliding window, and when data drift is detected in the model, retraining the fault prediction AIOps model based on a sliding window using a training set that only includes the identified data drift; Based on the benchmark model degradation indicators and performance evaluation indicators, the number of third feature changes of the fault prediction AIOps model during training and testing using the dataset is monitored. At the same time, the concept drift of the recognition model is detected through the concept drift detector and a drift alarm is issued.

2. The adaptive AIOps data drift detection method according to claim 1, characterized in that: The raw data is preprocessed to obtain data sets for model training and model testing, including: Obtain the original data for model training and testing, and clean the original data to remove noise and outliers; Apply standardization or normalization methods to the raw data after data cleaning to a unified scale, and convert the characteristic values ​​of the converted raw data into a distribution with a mean of 0 and a standard deviation of 1 to obtain normalized data. x std : in, x is the original eigenvalue, μ is the mean of the feature, σ is the standard deviation of the feature; For normalized data x std Missing value processing is performed to obtain a data set for model training and model testing, wherein the data set includes a training set and a test set.

Images