Architecture performance intelligent alarm and root analysis method based on big data analysis

Abstract

The invention discloses an architecture performance intelligent alarm and root analysis method based on big data analysis, and belongs to the technical field of artificial intelligence, and the method comprises the steps: obtaining monitoring data in a distributed architecture system, and carrying out the preprocessing; generating alarm event data through an alarm rule engine based on the preprocessed monitoring data, mining a frequent mode in the alarm event data by using an Apriori algorithm, and performing association rule analysis based on the frequent mode to obtain an alarm mode feature vector; splicing the alarm mode feature vector with the time sequence feature of the alarm event, inputting a splicing result into an LSTM model for prediction, obtaining an alarm event prediction trend feature, and splicing the alarm event prediction trend feature with the alarm mode fusion feature to obtain an alarm fusion feature; and classifying the alarm fusion features by using an alarm classification and root cause analysis model based on a random forest algorithm, and outputting an alarm category and a root cause. According to the invention, the accuracy and effectiveness of the alarm event are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence technology, more particularly to an architecture performance intelligent alarm and root cause analysis method based on big data analysis. BACKGROUND

[0002] With the rapid development of information technology, artificial intelligence technology has become the core driving force for promoting social progress and industrial transformation. As an important part of artificial intelligence technology, big data analysis has become increasingly important. Big data analysis involves collecting, processing and analyzing large and complex data sets to reveal hidden patterns, correlations, trends and other potential information in the data, becoming a key tool for enhancing competitiveness and innovation capability in various industries. In the power grid industry, big data analysis technology not only improves the safety, reliability and economy of the power grid, but also promotes the sustainable development of the smart grid big data industry.

[0003] Smart grid big data has the characteristics of large data volume, diverse sources, and complex formats, covering various aspects of power grid operation such as power generation, transmission, transformation, distribution, and power consumption. In particular, in the distributed system architecture, the monitoring and early warning of system performance face the challenges of large data size and high complexity. In this context, architecture performance monitoring tools play a crucial role. Tools such as SkyWalking can analyze the running status of the current architecture system when dealing with large amounts of complex data, and can alarm through alarm rules. It provides comprehensive performance monitoring and analysis capabilities for distributed systems, helping users optimize system performance and improve system stability and reliability.

[0004] However, SkyWalking and other tools have a significant impact on monitoring data and early warning based on preset rules, but they cannot perform correlation analysis and intelligent early warning, nor can they further analyze the root cause of the warning. First, the large amount of data collected by SkyWalking poses a challenge to the performance of analysis technology. Second, SkyWalking's alarm management function relies on user-configured alarm rules, which need to be manually set and lack the ability to adapt and learn themselves, limiting the implementation of intelligent early warning.

[0005] Analyzing the data monitored by tools such as SkyWalking is crucial for intelligent early warning of architecture performance. These data record the transfer of requests between various components by providing detailed tracking of request flow in distributed systems, which is crucial for identifying potential early warning sources. However, existing analysis methods have limitations, as they cannot deeply analyze these data from multiple dimensions and lack an intelligent early warning mechanism, making it difficult to intelligently perform early warning and traceability. Currently, most alarm systems mainly rely on single indicators or simple rules, which limits the accuracy of alarms and makes it difficult to distinguish between real performance problem alarms and false alarms caused by normal fluctuations. Therefore, in order to improve the accuracy and effectiveness of alarms, it is urgent to build an intelligent alarm mechanism based on multi-dimensional data correlation. SUMMARY

[0006] Therefore, the present application provides an architecture performance intelligent alarm and root cause analysis method based on big data analysis, which can improve the accuracy and effectiveness of alarms.

[0007] To achieve the above purpose, the present application adopts the following technical solutions:

[0008] An architecture performance intelligent alarm and root cause analysis method based on big data analysis, comprising the following steps:

[0009] Obtain monitoring data in a distributed architecture system and preprocess it; the monitoring data includes Trace data and Metrics data;

[0010] Based on the preprocessed monitoring data, generate alarm event data set through alarm rule engine;

[0011] Use Apriori algorithm to mine frequent item sets and strong association rules in alarm event data, encode the mined frequent item sets and strong association rules to obtain alarm pattern feature vector;

[0012] Splice the alarm pattern feature vector with the time series features of the alarm event, and input the spliced alarm splicing features into the LSTM prediction model to obtain alarm event prediction trend features;

[0013] Combine the alarm event prediction trend features and the alarm pattern feature vector to obtain alarm fusion features;

[0014] Use random forest algorithm to classify the alarm fusion features to obtain the class label and root cause label of the alarm event.

[0015] Further, the obtained monitoring data is preprocessed, specifically including,

[0016] The obtained Trace data and Metrics data are respectively preliminarily preprocessed, including data cleaning, data deduplication and unit unification processing;

[0017] The preliminarily preprocessed Trace data and Metrics data are spliced by taking "timestamp + TraceId / SegmentId + instance ID" as the alignment key to obtain spliced monitoring data;

[0018] The obtained spliced monitoring data is subjected to secondary data deduplication, data cleaning and normalization processing to obtain the final preprocessed monitoring data.

[0019] Further, the frequent item sets and strong association rules in the alarm event data are mined by using the Apriori algorithm, specifically including:

[0020] Each alarm event in the alarm event data set D is represented as an item set, and each item in the item set represents an alarm attribute of the alarm event;

[0021] The confidence of each single-item alarm attribute in the alarm event data set D is calculated:

[0022] Single-item frequent item sets are obtained by acquiring single-item alarm attributes with a confidence not lower than a minimum confidence threshold;

[0023] Based on the single-item frequent item sets, high-order candidate item sets are iteratively generated through connection and pruning operations;

[0024] The confidence of each candidate item set in the high-order candidate item set is calculated, and a new frequent item set is generated according to the set threshold until no new frequent item set can be generated, and the frequent item set Y representing the final frequent pattern is obtained.

[0025] According to the frequent item set Y, a plurality of candidate rules are obtained, and based on each candidate rule X=> (Y-X), the confidence thereof is calculated:

[0026] confidence(X=> (Y-X)) = support(Y) / support(X)

[0027] Wherein, X represents a non-empty subset in the frequent item set Y;

[0028] All rules with a confidence greater than or equal to a set threshold are screened out, which are the obtained strong association rules.

[0029] Further, the mined frequent item sets and strong association rules are encoded to obtain an alarm pattern feature vector, specifically including,

[0030] The strong association rule features mined by the Apriori algorithm are abstracted into a Boolean function, and based on the Boolean function, the rule activation degree at time t is defined:

[0031] aj(t)=support(Xj∪Yj,t) / support(Xj,t);

[0032] Encode the rule activation of all time into the alarm pattern feature vector.

[0033] Further, the alarm pattern feature vector is spliced with the time sequence feature of the alarm event, specifically including:

[0034] Resample the alarm event data according to a fixed time granularity, count the alarm count and type distribution information in each window, and generate the time sequence feature S(t) of the alarm event;

[0035] Splice the rule activation of multiple rules and the time sequence feature S(t) of the alarm event to obtain the spliced alarm splicing feature.

[0036] Further, the random forest algorithm is used to classify the alarm fusion feature to obtain the category label and root cause label of the alarm event, specifically including:

[0037] Random forest model training phase:

[0038] Encode each historical alarm into a high-dimensional feature vector covering time dimension, static dimension and alarm event prediction trend features, and use the manually labeled root cause category as a supervised label, and through decision tree ensemble learning, a trained random forest model is obtained;

[0039] Classification prediction phase:

[0040] The trained random forest model is used to classify the input alarm event data features to obtain the category label and root cause label of the alarm event.

[0041] Further, the above method further includes visualizing the obtained category label and root cause label of the alarm event.

[0042] On the other hand, the application also discloses an architecture performance intelligent alarm and root cause analysis system based on big data analysis, comprising a data acquisition module and a data processing module, the data acquisition module is used for acquiring monitoring data in a distributed architecture system and pre-processing, wherein the monitoring data includes Trace data and Metrics data; the data processing module is used for receiving the pre-processed monitoring data, and realizing the architecture performance intelligent alarm and root cause analysis method based on big data analysis of any one of the application.

[0043] Compared with the prior art, the technical scheme provided by the present application has the following beneficial effects:

[0044] The present application provides an architecture performance intelligent alarm and its alarm root analysis method based on big data analysis, aiming to improve the performance monitoring and alarm response capability of distributed system architecture. Firstly, the method integrates multi-dimensional data such as Trace and Metrics of service instances, realizes all-round monitoring of the performance of distributed system architecture, and ensures timely discovery and accurate early warning of performance problems. This all-round performance monitoring capability enables the system to capture potential performance bottlenecks from different angles, providing a solid data foundation for subsequent alarm event data extraction analysis and optimization.

[0045] The present application uses Apriori algorithm and association rule analysis to mine frequent patterns from historical alarm event data, identify complex relationships between alarms, significantly improve the accuracy of alarms, and accurately locate the problem root. This accurate alarm and root analysis capability greatly reduces false positives and false negatives, and improves the operation and maintenance efficiency.

[0046] In addition, by using time series analysis prediction and machine learning model, the present application can predict potential performance problems and issue early warnings, so that operation and maintenance personnel can take measures at the problem germination stage, effectively avoiding the spread of system failure.

[0047] The present application also designs an intelligent alarm mechanism, which can aggregate multiple alarms caused by the same root cause into one event, reduce alarm noise, and provide root cause analysis report. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical schemes in the embodiments of the present application or the prior art, the drawings needed in the embodiment or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present application, and those skilled in the art can obtain other drawings according to the provided drawings without creative labor.

[0049] Figure 1 The architecture performance intelligent alarm and root analysis method provided by the present application is shown in the whole flowchart. DETAILED DESCRIPTION

[0050] Clearly, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0051] The embodiments of the present application disclose an architecture performance intelligent alarm based on big data analysis and a method for alarm root analysis, and aim to improve the performance monitoring and alarm response capability of a distributed system architecture.

[0052] To achieve the above-mentioned application purposes, the present application adopts the following technical solutions:

[0053] The architecture performance intelligent alarm based on big data analysis and the method for alarm root analysis specifically include the following steps:

[0054] S1: collect and preprocess data using SkyWakling tools

[0055] Trace and Metrics data of service instances in a distributed system are collected by using monitoring tools such as SkyWalking. These data track the request flow in the system and record the transmission process of the request between various components, thereby providing detailed data basis for subsequent performance analysis.

[0056] Firstly, automatic collection of data is performed, and Trace and Metrics data in the application are automatically collected by SkyWalkingAgent and reported to a SkyWalkingOAP server. An example of a data reporting formula (TraceSegment reporting) is as follows:

[0057] Reported data = {TraceId, SegmentId, Spans, Logs,...}

[0058] Then, data storage and query are performed according to a data reporting protocol (HTTP protocol).

[0059] The SkyWalkingOAP server stores the received data in Elasticsearch. An example of a storage structure (ElasticsearchDocument) is as follows:

[0060] { / / json

[0061] "traceId":"traceId",

[0062] "segmentId":"segmentId",

[0063] "spans":[...],

[0064] "logs":[...]

[0065] }

[0066] Querying trace data and Metrics data through SkyWalking UI.

[0067] S2 Data storage and preprocessing

[0068] First, data storage: configure a distributed database, choose a distributed database HBase suitable for time series data and high-throughput writing, and configure database parameters to adapt to large-scale data storage needs.

[0069] Second, data preprocessing: mainly data cleaning, identifying and deleting missing values or incorrect data points; data deduplication, using data features (such as ID or timestamp) to identify and remove duplicates, deleting duplicate Trace or Metrics records, and retaining unique data points; data normalization, scaling data proportionally to fall within a small specific interval, facilitating comparison and processing; outlier detection, identifying and processing outliers or outliers in data, improving data quality, etc. Based on the preprocessed Trace / Metrics data, automatically identify abnormalities through a pre-set rules engine or anomaly detection model and generate structured alarm events, while associating original monitoring features and root cause labels, forming an alarm event dataset for subsequent Apriori, LSTM, and random forest modeling.

[0070] S3 Data analysis and modeling

[0071] First, use the historical alarm event data stored after S1 and S2 processing to mine frequent patterns and association rules in historical alarm event data. It should be noted that the alarm event data used in this step is not the original monitoring data, but the structured abnormal event records extracted from the monitoring data by the alarm event generation module defined in the S2 step. This record contains alarm types, associated monitoring features, and root cause labels.

[0072] Apriori algorithm is used to mine frequent patterns from historical alarm event data. Apriori algorithm is based on the principle that if an item set is frequent, then all its subsets must also be frequent. First, a minimum confidence threshold needs to be set, which is used to determine the minimum frequency of an item set appearing in the data set. Only item sets with a confidence not lower than this threshold are considered frequent item sets. For example, suppose we have a historical alarm event data set D, which contains multiple alarm events, each of which consists of several alarm attributes (such as alarm time, alarm type, involved components, etc.). The confidence of each single alarm attribute in the data set D is calculated, and the confidence calculation formula is:

[0073]

[0074] where X is a single alarm attribute, count(X) represents the number of times X appears in the data set D, and |D| is the total number of records in the data set D. All single alarm attributes with a confidence not lower than the minimum confidence threshold are found to form the frequent item set L1. Based on the frequent item set L1, candidate item sets C2 are generated through connection and pruning operations. The connection operation combines two different single alarm attributes in L1 into an item set, and the pruning operation removes item sets that do not meet the minimum confidence threshold. Repeat the above connection and pruning operations to generate candidate item sets, candidate item sets, etc., until no new frequent item set can be generated. The final frequent item set set contains the frequent patterns in the historical alarm event data. Based on the mined frequent patterns, association rule analysis is performed. For each frequent item set Y, the confidence of all non-empty subsets X is calculated, and the confidence calculation formula is:

[0075]

[0076] where Y-X represents the remaining part after removing the subset X from the frequent item set Y. By comparing the confidence with the set minimum confidence threshold, strong association rules are determined, which reveal the complex relationships between alarms and help identify alarm hot patterns and rules.

[0077] Second, based on the processed and time series stored data, a time series analysis model is used to predict the alarm event data generated based on monitoring data

[0078] Time series analysis models such as LSTM (Long Short-Term Memory) are used to predict alarm event data. First, the alarm event data generated from monitoring data is organized chronologically to form a time series dataset. For example, the number of alarms and the distribution of alarm types at each time point are used as sample points in the dataset. The dataset is preprocessed, including data normalization, mapping the data to a specific interval (e.g., [-1,1] or [0,1]) to improve the training efficiency and accuracy of the model. The normalization formula can be determined according to the specific normalization method, such as the min-max normalization formula:

[0079]

[0080] Where x represents the original data value, and min(x) and max(x) are the minimum and maximum values ​​in the dataset, respectively. The preprocessed dataset is divided into training and test sets, typically in a certain ratio (e.g., 70% training set, 30% test set). The LSTM model is trained using the training set. During training, the model learns the time-series patterns and trends in the input historical alarm event data, adjusting its internal parameters to minimize prediction error. The training process involves complex neural network calculations, including forward propagation to calculate predicted values ​​and calculating the mean squared error (MSE) of the loss function.

[0081]

[0082] Among them, y i For the true value, The predicted value is given by n, where n is the number of samples, and the value is updated using the backpropagation algorithm.

[0083] Model parameters. After training, the model is evaluated using a test set. Predictive performance is measured by comparing the difference between predicted and actual values ​​using metrics such as Root Mean Square Error (RMSE) and Mean Absolute Error (MAE). The trained LSTM model is then used to predict future alarm events, identifying anomalies in advance. For example, it can predict the trend of alarm quantity over a future period or the probability of specific alarm types occurring.

[0084] Finally, machine learning algorithms are used to train alarm classification and root cause analysis models. Random forest is applied to train the joint feature FR, which is obtained by concatenating the predicted probability sequence P output by the aforementioned TL and LSTM.

[0085] Machine learning algorithms such as random forest are used to train alarm classification and root cause analysis models. The first step involves collecting and organizing the dataset used for model training. This dataset should contain rich alarm information features, such as alarm time, alarm type, involved components, relevant performance metrics (e.g., CPU utilization, memory usage), and historical alarm processing results. Simultaneously, each sample is labeled with its corresponding alarm information.

[0086] The first step is to classify the alarms (e.g., normal alarms, critical alarms, performance bottleneck alarms, etc.) and identify possible root causes (e.g., network failures, software errors, etc.). The second step involves feature engineering the dataset, dividing it into training and test sets, similar to the partitioning method used in the LSTM model. The third step is to train the random forest model using the training set. During training, for each decision tree, a subset of samples (called the bootstrap sample set) is randomly drawn from the training set with replacement, and a subset of features is randomly selected to construct the node splitting conditions for the decision tree. This reduces the model's variance and improves its generalization ability. During the construction of each decision tree, recursively splitting is performed based on the feature values ​​of the samples until a stopping condition is met (e.g., the tree depth reaches its maximum, the number of samples in a node is too small, or the purity of a node reaches a certain threshold).

[0087] The fourth step is to evaluate the model using a test set. Evaluation metrics may include accuracy, recall, and F1 score, which measure the model's performance on alert classification and root cause analysis tasks. The accuracy formula is:

[0088]

[0089] Where TP (TruePositive) represents true positives, i.e., the number of samples that the model predicted as positive and actually were positive; TN (TrueNegative) represents true negatives, i.e., the number of samples that the model predicted as negative and actually were negative; FP (FalsePositive) represents false positives, i.e., the number of samples that the model predicted as positive but actually were negative; and FN (FalseNegative) represents false negatives, i.e., the number of samples that the model predicted as negative but actually were positive. The recall rate is calculated using the following formula:

[0090]

[0091] The F1 score is an indicator that takes into account both precision and recall, and its calculation formula is:

[0092]

[0093] in, Adjust model parameters (such as the number of decision trees, the depth of the trees, the proportion of feature selection, etc.) according to the evaluation results, retrain and evaluate until the model performance reaches a satisfactory level. The final trained random forest model can accurately classify alarms and analyze the root cause of the alarm based on the input alarm event data features, providing valuable decision support for operation and maintenance personnel.

[0094] S4 Intelligent Alarm and Root Cause Analysis

[0095] First, tool integration and data access

[0096] First, integrate the cross-platform data visualization tool Grafana with the data source that stores alarm event data and analysis results. Second, use Elasticsearch as the data storage (configured in S2) and configure the corresponding data source connection parameters in Grafana to ensure accurate data access. For example, in Grafana, add a data source, input the server address, port number, index name, etc. of Elasticsearch, and establish a connection. Third, extract the alarm event data and analysis result related fields that need to be displayed from the data source. These fields include but are not limited to alarm time, alarm level, alarm description, involved components, correlation rule analysis results, prediction trend information, alarm classification results, and preliminary root cause analysis information, etc.

[0097] Second, real-time monitoring of system performance and call situation visualization

[0098] Create a dashboard (Dashboard) in Grafana or Kibana and design different visualization panels (Panel) according to requirements. For system performance monitoring, you can create a line chart panel to show the trend of key performance indicators (such as CPU usage, memory occupancy, network traffic, etc.) over time. Query the data source to get real-time performance indicator data and present it in the form of a line chart. Operation and maintenance personnel can visually observe the dynamic changes of system performance and timely discover performance bottlenecks or abnormal fluctuations. For example, use time as the horizontal coordinate and performance indicator value as the vertical coordinate, update data points every certain time interval (such as 1 minute), and draw a continuous line chart.

[0099] For system call situation monitoring, you can use bar charts or pie charts to show the call frequency and call time distribution between different components or services. For example, use a bar chart to show the number of calls to each service interface within a certain time period. The height of the bar represents the number of calls. By comparing the heights of different bars, you can quickly understand which service interfaces are frequently called and which are less frequently called to discover potential call anomalies or hot issues.

[0100] Set appropriate refresh intervals to ensure that the dashboard updates in real-time, reflecting the current actual running state of the system. For example, set the refresh interval to 5 seconds or 10 seconds, so that the operation and maintenance personnel can obtain the latest system performance and call information in a timely manner.

[0101] Then, the setting of the abnormal situation alarm mechanism

[0102] Configure alarm rules in Grafana. According to the performance requirements and business characteristics of the system, set thresholds for each key performance indicator and alarm-related indicator. For example, for CPU usage, set an alarm when it exceeds 80% (adjustable according to actual situation); for the number of specific types of alarms, trigger an alarm when it exceeds a certain threshold within a certain time (such as 10 times per minute of a certain severity level). Choose alarm notification methods such as email, SMS, instant messaging tools (such as Slack), etc. When an abnormal situation occurs during system operation, i.e. when the pre-set alarm rule conditions are met, automatically trigger alarm notification. The alarm information should include detailed descriptions of the abnormal situation, such as alarm time, alarm level, alarm-triggering indicator value, related component information, and possible root cause analysis preliminary hints (based on the analysis results of the previous steps), so that operation and maintenance personnel can quickly locate the problem.

[0103] Finally, alarm aggregation and root cause analysis report generation

[0104] Implement alarm aggregation technology to aggregate alarms based on their root cause characteristics. You can analyze the attributes of the alarms (such as alarm type, involved components, time window when the alarm occurred, etc.) to determine whether they belong to multiple alarms caused by the same root cause. For example, if multiple components simultaneously produce "connection timeout" alarms within a short period of time (such as 1 minute), and these components all depend on the same network service, then these alarms can be aggregated into one event, reducing alarm noise.

[0105] When the alarms are aggregated, generate a root cause analysis report. The report details the common root cause of the aggregated alarms, including root cause information based on association rule analysis and machine learning model analysis. For example, the root cause analysis report may indicate that this aggregated alarm is due to a configuration error in the network service, and provide relevant evidence and analysis process, such as which alarm event data features based on monitoring data support this conclusion, and through which association rules or model predictions this problem was discovered, etc., to improve the rationality and accuracy of the response. Correlate the root cause analysis report with the aggregated alarm event, and provide a link to view the report in the visual display, making it easy for operation and maintenance personnel to obtain detailed root cause analysis information at any time, so that targeted measures can be taken to solve the problem and ensure the stable operation of the distributed system.

[0106] Implementation Case 1

[0107] For the convenience of those skilled in the art to understand and implement the present application, the present application will be further described in detail below in conjunction with examples, and the examples described herein are only used to illustrate and explain the present application, and are not a limitation on the present application.

[0108] S1 data collection

[0109] First, select SkyWalking tool

[0110] SkyWalking is selected as the monitoring tool for the power grid distributed architecture system, because it can adapt to the complex distributed environment of the power grid informatization system and comprehensively monitor various service instances in the system.

[0111] Then, data collection process

[0112] Deploy SkyWalkingAgent at each key node of the power grid informatization system, such as the generator set control unit of the power plant, the automation system of the substation, and the server of the power distribution management system. SkyWalkingAgent automatically collects Trace and Metrics data of service instances. In the power generation link, it traces the running state data of the generator set, including the real-time changes of parameters such as power generation power, voltage, current, and the interaction information between various components (such as engine, generator, cooling system, etc.) inside the generator set, records the transmission process of requests among these components. At the same time, Metrics data will collect performance indicators such as device CPU usage, memory occupancy, network communication delay, etc., to provide detailed information for system performance evaluation.

[0113] Finally, data reporting and storage

[0114] SkyWalkingAgent will collect data according to the established data reporting formula (such as {TraceId, SegmentId, Spans, Logs,...}), and report it to the SkyWalkingOAP server through the HTTP protocol. After receiving the data, the SkyWalkingOAP server stores it in Elasticsearch, and the storage structure is as follows (example):

[0115] json

[0116] {

[0117] "traceId":"traceId",

[0118] "segmentId":"segmentId",

[0119] "spans":[...],

[0120] "logs":[...]}

[0121] }

[0122] Through the SkyWalking UI, you can easily query the trace data and Metrics data for subsequent analysis.

[0123] S2 data storage and preprocessing

[0124] First, data storage

[0125] Configure the distributed database HBase, and optimize the parameters of HBase according to the characteristics of the power grid information system, such as large data volume and rapid data growth. For example, increase the number of RegionServers to improve the parallelism of data writing and reading, adjust the block size to adapt to the storage structure of power grid data, optimize the Memstore size to improve memory utilization, and ensure efficient storage of massive power grid operation data.

[0126] Second, data preprocessing, first, the original Trace and original Metrics are respectively executed for missing value cleaning, format checking and preliminary deduplication; then the fusion record is obtained according to the association of TraceId and timestamp, and the fusion record is further deduplicated, normalized and outlier removed. Specifically, the following steps are included:

[0127] Data cleaning: carefully check the missing values in the data, such as in the transmission line monitoring data, if the current value is missing at a certain time, mark the record as invalid and delete it. For data points with incorrect format, such as records with time stamp format not conforming to the power grid data standard, also delete them.

[0128] Data deduplication: use the unique identifier in the data (such as the combination of device number and timestamp) to identify duplicate Trace or Metrics records. When processing the Metrics data of the substation, if it is found that the device number of two records is the same and the timestamp difference is very small (within a certain error range), it is considered as a duplicate record, and one of them is retained and the other is deleted.

[0129] Data normalization: normalize each performance indicator in the Metrics data, for example

[0130] Map the generated power to the interval [0, 1]. Assuming that the generated power is 500 MW at a certain time, after the min-max normalization formula ( where x is the original data value, min(x) and max(x) are the minimum and maximum values in the data set, respectively) to obtain the normalized power generation values (assuming the minimum power generation value in the data set is OMW and the maximum power generation value is 1000 MW). This allows different indicators to be compared and analyzed on the same order of magnitude.

[0131] Outlier detection: Statistical-based methods are used in combination with the physical characteristics of power grid operation to detect outliers. For example, the mean and standard deviation of voltage data are calculated, and data points that deviate from the mean by more than a certain multiple of the standard deviation (e.g., 3 times the standard deviation) are considered outliers. At the same time, considering that the voltage of the power grid has a certain standard value within the normal operating range, if the voltage value exceeds the normal operating range (e.g., exceeds the rated voltage by ±10%), even if it does not exceed the statistical range of outliers, it is also considered an outlier. For outliers, further analysis is performed to determine the possible causes, such as sudden changes in power grid load, equipment failure, or interference.

[0132] S3: Data analysis and modeling

[0133] First, frequent pattern mining and association rule analysis in historical alarm event data

[0134] First, set the parameters, set the minimum confidence threshold to 0.3 and the minimum support threshold to 0.6. Assume that the historical alarm event data set contains 800 alarm records, each record contains attributes such as alarm time, alarm type (e.g., power generation equipment failure alarm, power transmission line overload alarm, distribution transformer overheating alarm, etc.), involved equipment or components (e.g., a certain power generator, a certain power transmission line, a certain transformer in a substation, etc.).

[0135] Second, use the Apriori algorithm to calculate the confidence of each single alarm attribute. For example, calculate the confidence of the "power generation equipment failure alarm" type in the data set. Assuming that "power generation equipment failure alarm" occurs 280 times, its confidence is satisfies the minimum confidence threshold and is added to the frequent item set L1. Based on the frequent item set L1, candidate set C2 is generated through connection and pruning operations. For example, combine "power generation equipment failure alarm" with "power transmission line overload alarm" into a candidate item set, then calculate its confidence, if the confidence is lower than the minimum confidence threshold, delete it from the candidate item set. Repeat this process until no new frequent item set can be generated, finally obtain the frequent item set collection, these frequent item sets contain the frequent patterns in the historical alarm event data.

[0136] Third step, association rule analysis, for each frequent item set Y, calculate the confidence of all its non-empty subsets X. For example, for the frequent item set {power equipment failure alarm, power transmission line overload alarm}, calculate its confidence. Suppose there are 180 records of "power equipment failure alarm" and "power transmission line overload alarm" appearing at the same time, and 280 records of "power equipment failure alarm" appearing alone, then the confidence is Satisfy the minimum confidence threshold, so "power equipment failure alarm → power transmission line overload alarm" is a strong association rule. By comparing the confidence with the set minimum confidence threshold, determine the strong association rule, encode the obtained frequent item set and its confidence as a pattern feature vector FP, as an additional input dimension of the subsequent LSTM model. These rules help to identify the potential relationship and pattern between alarms, such as power equipment failure may cause power transmission line overload, etc.

[0137] Then, based on the time series analysis model, the alarm event data is predicted. LSTM is used to model the time series of the fusion features TL, where TL is obtained by splicing the time statistics of the alarm events and the pattern feature vector FP output by Apriori.

[0138] First step, data organization and preprocessing: organize the alarm event data into time series data set according to time sequence, for example, take 15 minutes as time interval, count the number of alarms and the distribution of different types of alarms in each time interval as sample points. Normalize the data set, map the data to the interval [-1, 1], and the normalization formula used is (Where μ is the mean, σ is the standard deviation). Calculate the activation of each association rule in each time window as a continuous numerical value, and splice it with the alarm statistics features to form a continuous input vector of uniform dimension; divide the preprocessed data set into 70% training set and 30% test set.

[0139] Second step, build LSTM model: build a neural network model containing three LSTM layers and a fully connected layer. The number of hidden units of the LSTM layer is set to 64, 32 and 16 respectively, the activation function uses tanh, the Adam optimization algorithm is used, the learning rate is set to 0.001, and the loss function selects mean square error (MSE). The number of output nodes of the fully connected layer is determined according to the prediction target, for example, to predict the trend of alarm number change in the next 2 hours, the number of output nodes is 8 (because 2 hours contain 8 15-minute intervals).

[0140] Third step, model training and evaluation, use the training set to train the LSTM model, during the training process, the model learns the time series pattern and trend according to the input historical alarm event data, adjusts the internal parameters to minimize the prediction error. After training, use the test set to evaluate the model, calculate the root mean square error (RMSE) and mean absolute error (MAE) and other indicators. For example, after evaluation, the RMSE is 0.22, and the MAE is 0.16, indicating that the model has good prediction performance. Use the trained LSTM model to predict future alarm event data, for example, predict the trend of the number of alarms in the next 2 hours, provide early warning for operation and maintenance personnel, so as to take timely measures to deal with possible power grid failures.

[0141] Finally, the machine learning algorithm trains the alarm classification and root cause analysis model.

[0142] First step, data preparation, collect and organize the data set used to train the model, including alarm time, alarm type, involved devices or components, related performance indicators (such as power factor of power generation equipment, active power of transmission line, oil temperature of distribution transformer, etc.) and historical alarm processing results, etc. Rich features. At the same time, label the corresponding alarm category (such as emergency alarm, important alarm, general alarm, etc.) and possible root cause information (such as equipment aging, natural factor influence, human operation error, etc.) for each sample. For example, for an alarm record, if it is an insulation damage alarm caused by long-term exposure of the transmission line to harsh natural environment, the alarm category is labeled as "emergency alarm", and the root cause information is "natural factor influence (harsh environment leads to insulation aging)".

[0143] Second step, feature engineering processing: process the features in the data set, such as One-Hot Encoding for categorical features, and standardization processing for numerical features (make the mean value 0 and the standard deviation 1). Divide the processed data set into 70% training set and 30% test set according to the proportion.

[0144] Third step, Random Forest Model Training and Evaluation: Apply the Random Forest algorithm to train the alarm classification and root cause analysis models. Set the number of decision trees in the random forest to 120, the maximum depth of the tree to 12, and the feature selection ratio to 0.6 (i.e., randomly select 60% of the features for node splitting conditions when building a decision tree). During training, each decision tree randomly selects a portion of samples from the training set with replacement for training. According to the feature values of the samples, recursive splitting is performed until the stopping conditions are met (such as the depth of the tree reaching the maximum value, the number of samples in the node being too small, or the purity of the node reaching a certain threshold, etc.). After training, the test set is used to evaluate the model, and the accuracy, recall rate, and F1 value are calculated. For example, after evaluation, the alarm classification model has an accuracy of 0.88, a recall rate of 0.83, and an F1 value of 0.85, indicating that the model has good performance in alarm classification tasks. The root cause analysis model has an accuracy of 0.78, a recall rate of 0.73, and an F1 value of 0.75, which can provide valuable reference for root cause analysis for operation and maintenance personnel. According to the evaluation results, adjust the model parameters (such as the number of decision trees, the depth of the tree, the feature selection ratio, etc.), retrain and evaluate until the model performance reaches a satisfactory level.

[0145] S4 Intelligent Alarm and Root Cause Analysis

[0146] First, tool integration and data access

[0147] Integrate the Grafana tool with the Elasticsearch data source that stores alarm event data and analysis results. Add a data source in Grafana and input the server address, port number, index name, and other information of Elasticsearch to establish a connection. Extract the alarm event data and analysis result related fields that need to be displayed from the data source, including alarm time, alarm level, alarm description, involved devices or components, associated rule analysis results, prediction trend information, alarm classification results, and preliminary root cause analysis information, etc.

[0148] Second, real-time monitoring of system performance and call situation visualization

[0149] In Grafana, create a dashboard and design different visualization panels. For example, create a line chart panel to show the power generation trend of each power generation device over time. The horizontal axis represents time, and the vertical axis represents power generation values. Update data points every 5 minutes to allow operators to observe the stability of power generation equipment. Create a column chart panel to show the active power transmission of different transmission lines. The height of the column represents the active power size. By comparing the heights of different columns, quickly understand the load distribution of the transmission line and find potential overload problems. Set the refresh interval to 10 seconds to ensure that the dashboard can update data in real time and reflect the actual running state of the information system.

[0150] Then, set up the abnormal situation alarm mechanism

[0151] In Grafana, configure alarm rules according to the operation requirements and performance standards of the information system. Set thresholds for key performance indicators and alarm-related indicators. For example, set the CPU usage threshold of the server to 85%, and trigger an alarm when it exceeds this threshold. For network bandwidth utilization, set an alarm when it exceeds 80% of the rated bandwidth. Choose SMS and instant messaging tools (such as enterprise internal communication software) as alarm notification methods. When the system runs abnormally and meets the pre-set alarm rule conditions, automatically trigger alarm notification. Alarm information includes detailed abnormal situation description, such as alarm time, alarm level, triggering alarm indicator value, related service or component information, and possible root cause analysis preliminary prompt.

[0152] Finally, alarm aggregation and root cause analysis report generation

[0153] The alarm aggregation technology is implemented, for example, when alarms of service instances in multiple information systems occur at the same time within a short time (e.g., within 10 minutes), and the service instances belong to the same business module or calling link, the alarms are aggregated into one event by analyzing the attributes of the alarms (e.g., alarm type, involved components, time window of generating the alarm, etc.), and the alarms are determined to be caused by the same root cause, so as to reduce alarm noise. When the alarms are aggregated, a root cause analysis report is generated. The report details the common root cause of the aggregated alarms, for example, "the aggregated alarms are caused by the performance bottleneck of the core service in the business module, which leads to response delay and high memory occupancy of the downstream subsystems dependent on the service. The evidence is as follows: through the association rule analysis, it is found that the response delay alarms of the subsystems are related to the performance index exception of the core service; according to the performance index data, the response time of the core service abnormally increases before the alarms occur. It is suggested to check the resource configuration of the core service, optimize the code logic, and alleviate the performance bottleneck". The root cause analysis report is associated with the aggregated alarm event, an entry for viewing the report is provided in the visual display, and the detailed root cause analysis information is provided for the operation and maintenance personnel to obtain at any time, so as to take targeted measures to solve the problem and ensure the stable operation of the distributed architecture of the information system.

[0154] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0155] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for intelligent architecture performance alerting and root cause analysis based on big data analytics, characterized in that, Includes the following steps: Acquire monitoring data from the distributed architecture system and preprocess it; the monitoring data includes trace data and metric data. Based on the preprocessed monitoring data, an alarm event dataset is generated through the alarm rule engine; The Apriori algorithm is used to mine frequent itemsets and strong association rules in alarm event data. The mined frequent itemsets and strong association rules are then encoded to obtain alarm pattern feature vectors. The alarm pattern feature vector is concatenated with the time series features of the alarm event, and the concatenated alarm features are then input into the LSTM prediction model to obtain the alarm event prediction trend features. The alarm event prediction trend features and alarm pattern feature vectors are combined to obtain alarm fusion features; The random forest algorithm is used to classify alarm fusion features to obtain the category label and root cause label of alarm events.

2. The method for intelligent architecture performance alarm and root cause analysis based on big data analysis according to claim 1, characterized in that, The acquired monitoring data undergoes preprocessing, specifically including: The acquired trace data and metrics data are preprocessed, including data cleaning, data deduplication, and unit standardization. The pre-processed Trace data and Metrics data are concatenated using "timestamp + TraceId / SegmentId + instance ID" as the alignment key to obtain concatenated monitoring data. The obtained spliced ​​monitoring data is subjected to secondary data deduplication, data cleaning and normalization to obtain the preprocessed final monitoring data.

3. The method for intelligent architecture performance alarm and root cause analysis based on big data analysis according to claim 1, characterized in that, The Apriori algorithm is used to mine frequent itemsets and strong association rules in alarm event data, specifically including: Each alarm event in the alarm event dataset D is represented as an itemset, and each item in the itemset represents an alarm attribute of the alarm event; Calculate the confidence score of each individual alarm attribute in the alarm event dataset D: Collect individual alarm attributes with a confidence level not lower than the minimum confidence threshold and form them into frequent itemsets. Based on frequent single-item itemsets, higher-order candidate itemsets are iteratively generated through join and pruning operations. Calculate the confidence level of each candidate item set in the higher-order candidate item set, and generate new frequent itemsets according to the set threshold until no new frequent itemsets can be generated, thus obtaining the frequent itemset Y representing the final frequent pattern. Several candidate rules are obtained from the frequent itemset Y. For each candidate rule X => (YX), its confidence level is calculated: confidence(X=>(YX))=support(Y) / support(X) Where X represents a non-vacuum subset in a frequent itemset Y; The rules that are selected based on their confidence level being greater than or equal to the set threshold are the strong association rules.

4. The method for intelligent architecture performance alarm and root cause analysis based on big data analysis according to claim 1, characterized in that, The mined frequent itemsets and strong association rules are encoded to obtain alarm pattern feature vectors, specifically including... The strong association rule features mined by the Apriori algorithm are abstracted into Boolean functions. Based on these Boolean functions, the rule activation degree at time t is defined as follows: aj(t)=support(Xj∪Yj,t) / support(Xj,t); Encode the rule activation at all times into alarm pattern feature vectors.

5. The method for intelligent architecture performance alarm and root cause analysis based on big data analysis according to claim 4, characterized in that, The alarm pattern feature vector is concatenated with the time series features of the alarm event, specifically including: The alarm event data is resampled at a fixed time granularity, and the alarm count and type distribution information within each window are statistically analyzed to generate the time series feature S(t) of the alarm event; The rule activation degree of multiple rules and the time series feature S(t) of alarm events are concatenated to obtain the concatenated alarm feature.

6. The method for intelligent architecture performance alarm and root cause analysis based on big data analysis according to claim 1, characterized in that, The random forest algorithm is used to classify the alarm fusion features to obtain the category labels and root cause labels of the alarm events, specifically including: Random Forest Model Training Phase: Each historical alarm is encoded into a high-dimensional feature vector that simultaneously covers the time dimension, static dimension, and alarm event prediction trend features. The root cause category labeled by humans is used as the supervision label. Through decision tree ensemble learning, a well-trained random forest model is obtained. Classification prediction stage: The trained random forest model is used to classify the features of the input alarm event data to obtain the category label and root cause label of the alarm event.

7. The method for intelligent architecture performance alarm and root cause analysis based on big data analysis according to claim 1, characterized in that, It also includes a visual representation of the category and root cause labels of the alarm events.

8. A big data analytics-based intelligent architecture performance alarm and root cause analysis system, characterized in that, The system includes a data acquisition module and a data processing module. The data acquisition module is used to acquire monitoring data from the distributed architecture system and perform preprocessing, wherein the monitoring data includes trace data and metric data. The data processing module is used to receive the processed monitoring data and implement the intelligent alarm and root cause analysis method for architecture performance based on big data analysis as described in any one of claims 1-7.

Images