A multi-source monitoring data real-time fusion analysis method and device

Abstract

The application belongs to the technical field of IT operation and maintenance monitoring, and relates to a real-time fusion analysis method and device for multi-source monitoring data. The method aims at the problems of data islands caused by single-source analysis, low fault positioning efficiency, and many redundant alarms and difficult discovery of cross-device implicit correlations when the existing IT operation and maintenance monitoring system processes multi-source heterogeneous data. By constructing a multi-source data access adaptation matrix and a data feature extraction matrix, and establishing a real-time correlation analysis model, the method adapts to all-scene data collection characteristics, realizes efficient fusion and real-time analysis of multi-source heterogeneous monitoring data, responds to alarms quickly, and can accurately locate IT system fault points, thereby improving the intelligence and efficiency of operation and maintenance monitoring.

Description

Technical Field

[0001] This invention belongs to the field of IT operations and maintenance monitoring technology, specifically relating to a method and apparatus for real-time fusion analysis of multi-source monitoring data. Background Technology

[0002] Traditional IT operations and maintenance monitoring systems have two main drawbacks when processing heterogeneous data (metrics, logs, alarms, traffic) from multiple sources, such as servers and network devices: First, they use single-source independent analysis, resulting in data silos and requiring separate checks of different data sources for fault location, which is inefficient; second, they are prone to generating a large number of redundant alarms and false alarms, and cannot detect hidden anomalies across devices and metrics, leading to untimely fault detection.

[0003] In existing technologies, most multi-source data fusion methods employ complex machine learning algorithms for training on the entire dataset. While these methods can achieve some fusion effect, they are time-consuming to run and have poor real-time performance, making them unsuitable for the real-time monitoring needs of IT systems. Furthermore, these methods suffer from poor compatibility, making it difficult to adapt to various data collection methods such as Agent, SNMP, and API, thus hindering practical application. These are the shortcomings of existing technologies.

[0004] In view of this, it is very necessary to provide a real-time fusion analysis method and apparatus for multi-source monitoring data to solve the above-mentioned defects in the prior art. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing IT operation and maintenance monitoring data processing methods, such as inefficient fusion of multi-source heterogeneous data, low fault location efficiency, alarm redundancy, and poor real-time performance, by providing a real-time fusion analysis method and device for multi-source monitoring data to solve the aforementioned technical problems.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A real-time fusion analysis method for multi-source monitoring data includes the following steps: Step S1: Collect heterogeneous monitoring data from multiple sources in the IT system; Step S2: Standardize and parse heterogeneous monitoring data through a multi-source data access adaptation matrix to generate standard data frames in a unified format; perform timestamp synchronization calibration on the standard data frames. Step S3: Extract and filter the feature dimensions of the standard data frame using the data feature extraction matrix to obtain the fused feature set; Step S4: Construct a real-time correlation analysis model to perform spatiotemporal correlation matching on the fused feature set and filter out irrelevant redundant feature data. Identify abnormal association features from the associated feature data, count the number of abnormal association features, and set the association weight for each abnormal association feature; Step S5: Calculate the weighted total of abnormal correlation features, and generate a graded alarm threshold based on the weighted total of abnormal correlation features; If the weighted total of abnormal correlation features exceeds the preset hierarchical alarm threshold, the IT system is judged to be operating abnormally, and the alarm information is sent to the linked multi-platform alarm terminal, which includes a visualization platform, SMS, email, and enterprise-level social software. If the weighted total of abnormal correlation features does not exceed the hierarchical alarm threshold, the IT system is judged to be operating normally, and the process returns to step S1 to continue multi-source heterogeneous data collection and analysis.

[0007] Preferably, the heterogeneous monitoring data in step S1 specifically includes: Device metrics, log data, alarm data, and traffic data of servers, network devices, databases, and middleware in IT systems.

[0008] Preferably, the process of establishing the multi-source data access adaptation matrix in step S2 includes: Based on the format specifications of each monitoring data, the core parsing fields of each monitoring data are determined; A unified field mapping rule is set for each type of core parsing field to construct a conversion relationship from heterogeneous monitoring data to standard data frames; the conversion relationship can parse the collected heterogeneous monitoring data of various types. Construct a multi-source data access adaptation matrix based on the transformation relationship; The features in the standard data frame include: data type, data value, acquisition time, time interval, device identifier, device type, cluster to which the device belongs, indicator name, indicator threshold, indicator deviation rate, whether it is abnormal, abnormal level, and abnormal type.

[0009] Preferably, the data feature extraction matrix processing in step S3 includes: Identify the core features required for heterogeneous monitoring data fusion analysis; filter and retain the core features from standard data frames: data type, data value, acquisition time, device identifier, device type, indicator name, whether it is abnormal, abnormal level, and abnormal type; The filtered features and their corresponding data are structured and encapsulated to obtain a fused feature set; Each feature data in the fused feature set contains a unique device identifier and a timestamp identifier; After obtaining the fused feature set, the pre-trained anomaly recognition model in the IT system can be used to make a preliminary anomaly judgment on the feature data in the fused feature set, and the anomaly level and anomaly type can be marked for the feature data with anomalies, so as to achieve the preliminary screening of abnormal features.

[0010] Preferably, step S4 specifically includes: The feature data within the fused feature set are grouped according to the device identifier as the first dimension, and all feature data of the same device are grouped together. Then, the feature data within the fused feature set are grouped in a second dimension according to the collection timestamp, forming a spatiotemporal dimension group set; A real-time correlation analysis model is constructed, which includes preset spatiotemporal correlation rules; the real-time correlation analysis model is invoked to calculate the correlation degree of feature data within the same spatiotemporal dimension group set; a correlation degree threshold is set, feature data with a correlation degree higher than the preset correlation degree threshold is retained, and redundant feature data without correlation are filtered out; By calling the real-time correlation analysis model, feature data of devices with business correlations within different spatiotemporal dimension group sets are matched across groups, such as matching application servers with database servers, and matching switches with firewalls, to identify abnormal correlation features across devices and realize the discovery of hidden correlation faults. Count the number of abnormal correlation features and set the correlation weight for each abnormal correlation feature; The spatiotemporal correlation rules of the real-time correlation analysis model include time window correlation rules and topology correlation rules; wherein, the time window correlation rules are used to define the abnormal triggering relationship between different monitoring indicators within the same preset time window; the topology correlation rules are used to define the abnormal propagation relationship based on the business topology of the IT system.

[0011] Preferably, step S5 specifically includes: Calculate the weighted total of abnormal association features; If the weighted total of abnormal correlation features exceeds the preset hierarchical alarm threshold, the IT system is judged to be operating abnormally. Features with consistent device identifiers among the abnormal correlation features are aggregated to locate the associated devices of the abnormal fault; an emergency alarm is triggered and the fault handling process is linked to send alarms to alarm terminals on multiple platforms to achieve rapid fault response. If the weighted total of abnormal correlation features does not exceed the graded alarm threshold, the IT system is judged to be operating normally, and the process returns to step S1. The preset hierarchical alarm thresholds for the abnormal correlation features are divided according to the importance level of the IT system. For example, the first-level alarm threshold for core business systems is 8 and the second-level alarm threshold is 5, while the first-level alarm threshold for ordinary business systems is 15 and the second-level alarm threshold is 10, so as to achieve accurate alarm classification and avoid redundant alarms.

[0012] Furthermore, the present invention also provides a real-time fusion analysis device for multi-source monitoring data, comprising: The multi-source data acquisition module includes: Collect heterogeneous monitoring data from multiple sources in the IT system; The standardized parsing module contains: Heterogeneous monitoring data is standardized and parsed through a multi-source data access adaptation matrix to generate standard data frames in a unified format; the standard data frames are then time-stamped and calibrated. The feature extraction module contains: The calibrated standard data frame is subjected to feature extraction and filtering using a data feature extraction matrix to obtain a fused feature set. The correlation analysis module contains: Construct a real-time correlation analysis model to perform spatiotemporal correlation matching on the fused feature set and filter out irrelevant redundant feature data; Identify abnormal association features from the associated feature data, count the number of abnormal association features, and set the association weight for each abnormal association feature; The alarm output module contains: Calculate the weighted total of abnormal correlation features, and generate a graded alarm threshold based on the weighted total of abnormal correlation features; If the weighted total of abnormal correlation features exceeds the preset hierarchical alarm threshold, the IT system is judged to be operating abnormally, and the alarm information is sent to the linked multi-platform alarm terminal, which includes a visualization platform, SMS, email, and enterprise-level social software. If the weighted total of abnormal correlation features does not exceed the tiered alarm threshold, the IT system is judged to be operating normally, and multi-source heterogeneous data collection and analysis will continue.

[0013] Preferably, the heterogeneous monitoring data in the multi-source data acquisition module specifically includes: Device metrics, log data, alarm data, and traffic data of servers, network devices, databases, and middleware in IT systems; The heterogeneous monitoring data is collected by the built-in Agent collection unit, SNMP collection unit, API collection unit, and log collection unit. The Agent collection unit is used to collect the device indicator data of the server in the IT system, the SNMP collection unit is used to collect the traffic data of the network device in the IT system, the API collection unit is used to collect the operating status data of the database and middleware in the IT system, and the log collection unit is used to collect the log data and alarm data of each device in the IT system.

[0014] Preferably, the process of establishing the multi-source data access adaptation matrix in the standardized parsing module includes: Based on the format specifications of each monitoring data, the core parsing fields of each monitoring data are determined; A unified field mapping rule is set for each type of core parsing field to construct a conversion relationship from heterogeneous monitoring data to standard data frames; the conversion relationship can parse the collected heterogeneous monitoring data of various types. Construct a multi-source data access adaptation matrix based on the transformation relationship; The features in the standard data frame include: data type, data value, acquisition time, time interval, device identifier, device type, cluster to which the device belongs, indicator name, indicator threshold, indicator deviation rate, whether it is abnormal, abnormal level, and abnormal type.

[0015] Preferably, the data feature extraction matrix processing procedure in the feature extraction module includes: Identify the core features required for heterogeneous monitoring data fusion analysis; filter and retain the core features from standard data frames: data type, data value, acquisition time, device identifier, device type, indicator name, whether it is abnormal, abnormal level, and abnormal type; The filtered features and their corresponding data are structured and encapsulated to obtain a fused feature set; Each feature data in the fused feature set contains a unique device identifier and a timestamp identifier.

[0016] After obtaining the fused feature set, a pre-trained anomaly detection model from the IT system can be used to perform preliminary anomaly judgment on the feature data in the fused feature set. Anomaly levels and types are labeled for the feature data exhibiting anomalies, thus achieving preliminary screening of anomalous features. Preferably, the correlation analysis module specifically includes: The feature data within the fused feature set are grouped according to the device identifier as the first dimension, and all feature data of the same device are grouped together. Then, the feature data within the fused feature set are grouped in a second dimension according to the collection timestamp, forming a spatiotemporal dimension group set; A real-time correlation analysis model is constructed, which includes preset spatiotemporal correlation rules; the real-time correlation analysis model is invoked to calculate the correlation degree of feature data within the same spatiotemporal dimension group set; a correlation degree threshold is set, feature data with a correlation degree higher than the preset correlation degree threshold is retained, and redundant feature data without correlation are filtered out; By calling the real-time correlation analysis model, feature data of devices with business correlations within different spatiotemporal dimension group sets are matched across groups, such as matching application servers with database servers, and matching switches with firewalls, to identify abnormal correlation features across devices and realize the discovery of hidden correlation faults. Count the number of abnormal correlation features and set the correlation weight for each abnormal correlation feature; The spatiotemporal correlation rules of the real-time correlation analysis model include time window correlation rules and topology correlation rules; wherein, the time window correlation rules are used to define the abnormal triggering relationship between different monitoring indicators within the same preset time window; the topology correlation rules are used to define the abnormal propagation relationship based on the business topology of the IT system.

[0017] Preferably, the alarm output module specifically includes: Calculate the weighted total of abnormal association features; If the weighted total of abnormal correlation features exceeds the preset hierarchical alarm threshold, the IT system is judged to be operating abnormally. Features with consistent device identifiers among the abnormal correlation features are aggregated to locate the associated devices of the abnormal fault; an emergency alarm is triggered and the fault handling process is linked to send alarms to alarm terminals on multiple platforms to achieve rapid fault response. If the weighted total of abnormal correlation features does not exceed the graded alarm threshold, the IT system is judged to be operating normally, and multi-source heterogeneous data collection and analysis will continue. The preset hierarchical alarm thresholds for the abnormal correlation features are divided according to the importance level of the IT system. For example, the first-level alarm threshold for core business systems is 8 and the second-level alarm threshold is 5, while the first-level alarm threshold for ordinary business systems is 15 and the second-level alarm threshold is 10, so as to achieve accurate alarm classification and avoid redundant alarms.

[0018] The beneficial effects of this invention are as follows: First, by constructing a multi-source data access adaptation matrix, standardized parsing of heterogeneous data from device metrics, logs, alarms, and traffic collected through various methods such as Agent, SNMP, and API is achieved. This breaks down data silos, lays the foundation for multi-source data fusion analysis, and the parsing process is highly efficient and fast, adapting to the real-time monitoring needs of IT systems. Second, through a data feature extraction matrix and a real-time correlation analysis model, feature fusion and spatiotemporal correlation analysis of multi-source monitoring data are realized. This enables the identification of abnormal correlation features across devices and metrics, timely detection of hidden correlation faults in IT systems, and allows maintenance personnel to grasp the operating status of IT systems from a global perspective. Compared with traditional single-source data independent analysis methods, fault location has higher accuracy and... Efficiency; Furthermore, by filtering out redundant feature data and setting tiered alarm thresholds, the problem of redundant and false alarms in traditional monitoring systems is solved, making alarm information more accurate and reducing ineffective work for maintenance personnel. At the same time, the tiered alarm linkage with the fault handling process enables rapid fault response, realizing a shift from passive fault handling to proactive anomaly early warning, and improving the intelligence level of IT operation and maintenance monitoring. Finally, the method of this invention does not require complex full-data machine learning training, runs fast, and can complete the fusion analysis and alarm output of multi-source data in milliseconds. Moreover, it can be seamlessly integrated with the existing IT operation and maintenance monitoring architecture, making it easy to implement and apply.

[0019] Therefore, it is evident that the present invention has outstanding substantive features and significant progress compared with the prior art, and the beneficial effects of its implementation are also obvious. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0021] Figure 1 This is a flowchart of a real-time fusion analysis method for multi-source monitoring data provided by the present invention.

[0022] Figure 2 This is a schematic diagram of a real-time fusion analysis device for multi-source monitoring data provided by the present invention.

[0023] Among them, 1-multi-source data acquisition module, 2-standardization parsing module, 3-feature extraction module, 4-correlation analysis module, and 5-alarm output module. Detailed Implementation

[0024] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. The following embodiments are explanations of the present invention, but the present invention is not limited to the following implementation methods.

[0025] Example 1: like Figure 1 As shown, the IT operation and maintenance monitoring system provided in this embodiment can collect data from IT full-stack resources through multiple methods such as Agent, SNMP, API, and logs. The collected data types cover device indicators such as server CPU, memory, and disk, traffic data of firewalls and switches, and heterogeneous data such as operation logs and alarm information of various IT systems. A real-time fusion analysis method for multi-source monitoring data includes the following steps: Step S1: Collect heterogeneous monitoring data from multiple sources in the IT system; The heterogeneous monitoring data in step S1 specifically includes: Device metrics, log data, alarm data, and traffic data of servers, network devices, databases, and middleware in IT systems; During normal operation and maintenance monitoring, the IT operation and maintenance monitoring system uses multi-source data acquisition module 1 to collect data on the full stack resources of the IT system 24 / 7, collecting heterogeneous monitoring data such as device indicators, logs, alarms, and traffic, and transmitting the raw data to the standardized parsing module 2 in real time.

[0026] Step S2, the standardization and parsing step, in which: Heterogeneous monitoring data is standardized and parsed by a multi-source data access adaptation matrix to generate standard data frames in a unified format. The timestamps of the standard data frames are synchronized and calibrated to unify the timestamps of all data frames to the millisecond level, so that the time characteristics of standard data frames from different acquisition methods are consistent and analysis errors caused by acquisition delay are avoided. The process of establishing the multi-source data access adaptation matrix in step S2 includes: Based on the format specifications of each monitoring data, the core parsing fields for device metrics are determined to be device IP, collection time, metric name, and metric value; the core parsing fields for log data are device IP, log time, log level, and log content; the core parsing fields for alarm data are alarm device, alarm time, alarm type, and alarm level; and the core parsing fields for traffic data are device port, collection time, inbound traffic, and outbound traffic. A unified field mapping rule is set for each type of core parsing field. For example, device IP and alarm device are uniformly mapped to "device identifier", and collection time, log time, and alarm time are uniformly mapped to "collection time". This constructs a conversion relationship between heterogeneous monitoring data and standard data frames. The conversion relationship can parse the heterogeneous monitoring data of various types collected. Construct a multi-source data access adaptation matrix based on the transformation relationship.

[0027] The features in the standard data frame include: data type, data value, acquisition time, time interval, device identifier, device type, cluster to which the device belongs, indicator name, indicator threshold, indicator deviation rate, whether it is abnormal, abnormal level, and abnormal type.

[0028] Step S3: Extract and filter the feature dimensions of the standard data frame using the data feature extraction matrix to obtain the fused feature set; Step S3 specifically includes: Identify the core features required for heterogeneous monitoring data fusion analysis; filter and retain the core features from standard data frames: data type, data value, acquisition time, device identifier, device type, indicator name, whether it is abnormal, abnormal level, and abnormal type; The filtered features and their corresponding data are structured and encapsulated to obtain a fused feature set; Each feature data in the fused feature set contains a unique device identifier and a timestamp identifier; After obtaining the fused feature set, the pre-trained anomaly recognition model in the IT system can be used to make a preliminary anomaly judgment on the feature data in the fused feature set, and the anomaly level and anomaly type can be marked for the feature data with anomalies, so as to achieve the preliminary screening of abnormal features.

[0029] The anomaly identification model includes a base predictor, an anomaly discriminator, and a graded classifier. The base predictor employs multiple parallel time-series prediction units, each built on a long short-term memory network, to perform time-series predictions of key performance indicators such as CPU utilization, memory usage, and response time, outputting the predicted values ​​of the indicators at the next time step. The anomaly discriminator compares the predicted values ​​of the base predictor with the actual collected values, calculates the deviation rate, and triggers an anomaly labeling when the deviation rate exceeds a preset threshold. The graded classifier uses a gradient boosting decision tree model, taking the deviation rate, historical anomaly frequency, and indicator importance as inputs, to classify anomalies into three preliminary judgment levels: warning, moderately severe, and extremely severe, and identifies the anomaly type based on log keyword matching and an alarm rule base.

[0030] The anomaly detection model is trained using a combination of offline pre-training and online fine-tuning. In the offline pre-training phase, historical monitoring data is collected to construct a training sample set. For the base predictor, supervised learning is employed, using historical time series data as input and future time-series indicators as labels, with mean squared error as the loss function. For the ranking classifier, historical anomaly samples and their labeled anomaly levels and types are used as training data, and cross-entropy loss is applied. In the online fine-tuning phase, real-time alarm confirmation results are used to incrementally update the anomaly detection model parameters using an online learning algorithm. For the base predictor, backpropagation is performed to fine-tune the model based on the error between the actual and predicted indicator values. For the ranking classifier, the classification boundary is adjusted based on the anomaly levels and types corrected by maintenance personnel, continuously improving the model's accuracy and adaptability.

[0031] Step S4: Construct a real-time correlation analysis model to perform spatiotemporal correlation matching on the fused feature set and filter out irrelevant redundant feature data. Identify abnormal association features from the associated feature data, count the number of abnormal association features, and set the association weight for each abnormal association feature; Step S4 specifically includes: The feature data within the fused feature set are grouped according to the device identifier as the first dimension, and all feature data of the same device are grouped together. Then, the feature data within the fused feature set are grouped in a second dimension according to the collection timestamp, forming a spatiotemporal dimension group set; A real-time correlation analysis model is constructed, which includes preset spatiotemporal correlation rules; the real-time correlation analysis model is invoked to perform intra-group correlation matching: the correlation degree of feature data in the same spatiotemporal dimension group set is calculated, the correlation degree threshold is set to 0.7, feature data with correlation degree higher than the preset correlation degree threshold is retained, and redundant feature data without correlation are filtered out. Call the real-time correlation analysis model to perform cross-group correlation matching: perform cross-group matching on the feature data of devices with business correlation within different spatiotemporal dimension group sets, such as application server and database server, switch and firewall, to identify abnormal correlation features across devices and realize the discovery of hidden correlation faults; Feature data whose deviation from historical normal operation features is less than 5% in the fused feature set will be directly filtered out as invalid and redundant data to further improve analysis efficiency; From the associated feature data, extract the abnormal association features labeled with abnormality identifiers, and perform K-means clustering on the abnormal association features according to the abnormality type to obtain feature clusters of the same type of abnormality such as CPU over-limit, insufficient memory, and log error, thereby realizing automatic classification of abnormality types; Count the number of each abnormal correlation feature and set the correlation weight according to the abnormality level: Level 1 abnormality (severe) has a weight of 3, Level 2 abnormality (moderate) has a weight of 2, and Level 3 abnormality (minor) has a weight of 1. Calculate the weighted total of abnormal correlation features.

[0032] The spatiotemporal correlation rules of the real-time correlation analysis model include: time window correlation rules and topology correlation rules. The time window correlation rules define the anomaly triggering relationship between different monitoring indicators within the same preset time window. For example, if the database connection pool occupancy rate exceeds 90% for three consecutive collection cycles, and the application server's request response time exceeds 500ms within the same time window, then a temporal correlation is determined between the two. The topology correlation rules define the anomaly propagation relationship based on the IT system's business topology. For example, if the backend server health check of the load balancer fails, all network path devices between the load balancer and the corresponding failed backend server are included in the correlation analysis scope to identify the anomaly propagation path.

[0033] The real-time correlation analysis model comprises a data access layer, a feature processing layer, a correlation calculation layer, a rule engine layer, and a decision output layer. The data access layer is responsible for reading feature data from the fused feature set in real time, supporting streaming data access and flow control. The feature processing layer performs windowing processing on the input feature data, supporting three types: sliding window, scrolling window, and session window, enabling feature data caching management and fast retrieval. The correlation calculation layer includes a time series similarity calculation unit, employing a dynamic time warping algorithm to calculate the similarity between different indicator sequences; an anomaly propagation path tracking unit, performing a breadth-first search based on the IT system topology to identify anomaly propagation paths; and a correlation strength quantification unit, using an improved Pearson correlation coefficient and mutual information entropy to calculate the degree of correlation between features. The rule engine layer uses the RETE algorithm for rule matching, ensuring matching efficiency under large-scale rules. The decision output layer is responsible for outputting the correlated feature data and marking abnormal correlation features.

[0034] The real-time correlation analysis model is obtained through online training using reinforcement learning: based on the alarm handling results of real-time feedback, the confidence weight of spatiotemporal correlation rules is dynamically adjusted through reinforcement learning algorithm, and rules with frequent false alarms are automatically downgraded or disabled, thereby continuously optimizing the accuracy and adaptability of spatiotemporal correlation rules.

[0035] Step S5, the alarm output step, in which: Calculate the weighted total of abnormal correlation features, and generate a graded alarm threshold based on the weighted total of abnormal correlation features; If the weighted total of abnormal correlation features exceeds the preset hierarchical alarm threshold, the IT system is judged to be operating abnormally, and alarm information is sent to the linked visualization platform, SMS, email, and enterprise-level social software alarm terminals to achieve multi-terminal alarm reach. If the weighted total of abnormal correlation features does not exceed the hierarchical alarm threshold, the IT system is judged to be operating normally, and the process returns to step S1 to continue multi-source heterogeneous data collection and analysis.

[0036] Step S5 specifically includes: Calculate the weighted total of abnormal association features; The weighted total of abnormal correlation features is compared with the preset alarm thresholds, which are divided according to the importance level of the IT system: the first-level alarm threshold for core business systems is 8, and the second-level alarm threshold is 5; the first-level alarm threshold for ordinary business systems is 15, and the second-level alarm threshold is 10. If the weighted total of abnormal correlation features does not exceed the graded alarm threshold, the IT system is judged to be operating normally, and the process returns to step S1. If the weighted total of abnormal correlation features exceeds the preset hierarchical alarm threshold, the IT system is judged to be malfunctioning. Features with consistent device identifiers among the abnormal correlation features are aggregated to locate the associated devices of the abnormal fault. An emergency alarm is triggered and the fault handling process is linked, sending alarms to alarm terminals on multiple platforms to achieve rapid fault response. If the weighted total of abnormal correlation features exceeds the secondary alarm threshold but does not exceed the primary alarm threshold, a yellow alarm prompt is triggered on the visual platform alarm terminal, and alarm information is pushed to the enterprise-level social software alarm terminal of the maintenance personnel. If the weighted total of abnormal correlation features exceeds the primary alarm threshold, a red emergency alarm prompt is triggered on the visual platform alarm terminal, and alarm information is pushed to the maintenance personnel through SMS alarm terminal, email alarm terminal, and enterprise-level social software alarm terminal. The fault handling process of the IT maintenance monitoring system is linked, automatically creating a fault work order and assigning it to the corresponding maintenance personnel. The maintenance personnel locate the problem based on the faulty device and fault type in the alarm information, perform fault handling, clear the alarm information after handling, and restore normal monitoring and analysis.

[0037] Example 2: like Figure 2 As shown in the figure, this embodiment provides a real-time fusion analysis device for multi-source monitoring data, comprising: Multi-source data acquisition module 1, in which: Collect heterogeneous monitoring data from multiple sources in the IT system; The heterogeneous monitoring data in the multi-source data acquisition module 1 specifically includes: Device metrics, log data, alarm data, and traffic data of servers, network devices, databases, and middleware in IT systems; The heterogeneous monitoring data is collected by the built-in Agent collection unit, SNMP collection unit, API collection unit, and log collection unit. The Agent collection unit is used to collect the device indicator data of the server in the IT system, the SNMP collection unit is used to collect the traffic data of the network device in the IT system, the API collection unit is used to collect the operating status data of the database and middleware in the IT system, and the log collection unit is used to collect the log data and alarm data of each device in the IT system.

[0038] Standardized parsing module 2, in which: Heterogeneous monitoring data is standardized and parsed through a multi-source data access adaptation matrix to generate standard data frames in a unified format; the standard data frames are then time-stamped and calibrated. The process of establishing the multi-source data access adaptation matrix in the standardized parsing module 2 includes: Based on the format specifications of each monitoring data, the core parsing fields of each monitoring data are determined; A unified field mapping rule is set for each type of core parsing field to construct a conversion relationship from heterogeneous monitoring data to standard data frames; the conversion relationship can parse the collected heterogeneous monitoring data of various types. Construct a multi-source data access adaptation matrix based on the transformation relationship.

[0039] The features in the standard data frame include: data type, data value, acquisition time, time interval, device identifier, device type, cluster to which the device belongs, indicator name, indicator threshold, indicator deviation rate, whether it is abnormal, abnormal level, and abnormal type.

[0040] Feature extraction module 3, in which: The calibrated standard data frame is subjected to feature extraction and filtering using a data feature extraction matrix to obtain a fused feature set. The data feature extraction matrix processing procedure in the feature extraction module 3 includes: Identify the core features required for heterogeneous monitoring data fusion analysis; filter and retain the core features from standard data frames: data type, data value, acquisition time, device identifier, device type, indicator name, whether it is abnormal, abnormal level, and abnormal type; The filtered features and their corresponding data are structured and encapsulated to obtain a fused feature set; Each feature data in the fused feature set contains a unique device identifier and a timestamp identifier; After obtaining the fused feature set, the pre-trained anomaly recognition model in the IT system can be used to make a preliminary anomaly judgment on the feature data in the fused feature set, and the anomaly level and anomaly type can be marked for the feature data with anomalies, so as to achieve the preliminary screening of abnormal features.

[0041] Association analysis module 4, in which: Construct a real-time correlation analysis model to perform spatiotemporal correlation matching on the fused feature set and filter out irrelevant redundant feature data; Identify abnormal association features from the associated feature data, count the number of abnormal association features, and set the association weight for each abnormal association feature; The aforementioned correlation analysis module 4 specifically includes: The feature data within the fused feature set are grouped according to the device identifier as the first dimension, and all feature data of the same device are grouped together. Then, the feature data within the fused feature set are grouped in a second dimension according to the collection timestamp, forming a spatiotemporal dimension group set; A real-time correlation analysis model is constructed, which includes preset spatiotemporal correlation rules; the real-time correlation analysis model is invoked to calculate the correlation degree of feature data within the same spatiotemporal dimension group set; a correlation degree threshold is set, feature data with a correlation degree higher than the preset correlation degree threshold is retained, and redundant feature data without correlation are filtered out; By calling the real-time correlation analysis model, feature data of devices with business correlations within different spatiotemporal dimension group sets are matched across groups, such as matching application servers with database servers, and matching switches with firewalls, to identify abnormal correlation features across devices and realize the discovery of hidden correlation faults. Count the number of abnormal correlation features and set the correlation weight for each abnormal correlation feature; The spatiotemporal correlation rules of the real-time correlation analysis model include time window correlation rules and topology correlation rules; wherein, the time window correlation rules are used to define the abnormal triggering relationship between different monitoring indicators within the same preset time window; the topology correlation rules are used to define the abnormal propagation relationship based on the business topology of the IT system.

[0042] Alarm output module 5, in which: Calculate the weighted total of abnormal correlation features, and generate a graded alarm threshold based on the weighted total of abnormal correlation features; If the weighted total of abnormal correlation features exceeds the preset hierarchical alarm threshold, the IT system is judged to be operating abnormally, and the alarm information is sent to the linked multi-platform alarm terminal, which includes a visualization platform, SMS, email, and enterprise-level social software. If the weighted total of abnormal correlation features does not exceed the tiered alarm threshold, the IT system is judged to be operating normally, and multi-source heterogeneous data collection and analysis will continue.

[0043] The alarm output module 5 specifically includes: Calculate the weighted total of abnormal association features; If the weighted total of abnormal correlation features exceeds the preset hierarchical alarm threshold, the IT system is judged to be operating abnormally. Features with consistent device identifiers among the abnormal correlation features are aggregated to locate the associated devices of the abnormal fault; an emergency alarm is triggered and the fault handling process is linked to send alarms to alarm terminals on multiple platforms to achieve rapid fault response. If the weighted total of abnormal correlation features does not exceed the graded alarm threshold, the IT system is judged to be operating normally, and multi-source heterogeneous data collection and analysis will continue. The preset hierarchical alarm thresholds for the abnormal correlation features are divided according to the importance level of the IT system. For example, the first-level alarm threshold for core business systems is 8 and the second-level alarm threshold is 5, while the first-level alarm threshold for ordinary business systems is 15 and the second-level alarm threshold is 10, so as to achieve accurate alarm classification and avoid redundant alarms.

[0044] The above-disclosed embodiments are merely preferred embodiments of the present invention, but the present invention is not limited thereto. Any non-creative variations that can be conceived by those skilled in the art, as well as any improvements and modifications made without departing from the principles of the present invention, should fall within the protection scope of the present invention.

Claims

1. A method for real-time fusion analysis of multi-source monitoring data, characterized in that, Includes the following steps: Step S1: Collect heterogeneous monitoring data from multiple sources in the IT system; Step S2: Standardize and parse heterogeneous monitoring data through a multi-source data access adaptation matrix to generate standard data frames in a unified format; Perform timestamp synchronization calibration on standard data frames; Step S3: Extract and filter features from the calibrated standard data frame using the data feature extraction matrix to obtain a fused feature set; Step S4: Construct a real-time correlation analysis model to perform spatiotemporal correlation matching on the fused feature set and filter out irrelevant redundant feature data. Identify abnormal association features from the associated feature data, count the number of abnormal association features, and set the association weight for each abnormal association feature; Step S5: Calculate the weighted total of abnormal correlation features, and generate a graded alarm threshold based on the weighted total of abnormal correlation features; If the weighted total of abnormal correlation features exceeds the preset hierarchical alarm threshold, the IT system is judged to be operating abnormally, and the alarm information is sent to the linked multi-platform alarm terminal. If the weighted total of abnormal correlation features does not exceed the hierarchical alarm threshold, the IT system is judged to be operating normally, and the process returns to step S1 to continue multi-source heterogeneous data collection and analysis.

2. The real-time fusion analysis method for multi-source monitoring data according to claim 1, characterized in that, The heterogeneous monitoring data in step S1 specifically includes: Device metrics, log data, alarm data, and traffic data of servers, network devices, databases, and middleware in IT systems.

3. The real-time fusion analysis method for multi-source monitoring data according to claim 1, characterized in that, The process of establishing the multi-source data access adaptation matrix in step S2 includes: Based on the format specifications of each monitoring data, the core parsing fields of each monitoring data are determined; A unified field mapping rule is set for each type of core parsing field to construct a conversion relationship from heterogeneous monitoring data to standard data frames; the conversion relationship can parse the collected heterogeneous monitoring data of various types. Construct a multi-source data access adaptation matrix based on the transformation relationship; The features in the standard data frame include: data type, data value, acquisition time, time interval, device identifier, device type, cluster to which the device belongs, indicator name, indicator threshold, indicator deviation rate, whether it is abnormal, abnormal level, and abnormal type.

4. The real-time fusion analysis method for multi-source monitoring data according to claim 1, characterized in that, The data feature extraction matrix processing in step S3 includes: Identify the core features required for heterogeneous monitoring data fusion analysis; filter and retain the core features from standard data frames: data type, data value, acquisition time, device identifier, device type, indicator name, whether it is abnormal, abnormal level, and abnormal type; The filtered features and their corresponding data are structured and encapsulated to obtain a fused feature set; Each feature data in the fused feature set contains a unique device identifier and a timestamp identifier.

5. The real-time fusion analysis method for multi-source monitoring data according to claim 1, characterized in that, Step S4 specifically includes: The feature data within the fused feature set are grouped according to the device identifier as the first dimension, and all feature data of the same device are grouped together. Then, the feature data within the fused feature set are grouped in a second dimension according to the collection time, forming a spatiotemporal dimension group set; A real-time correlation analysis model is constructed, which includes preset spatiotemporal correlation rules; the real-time correlation analysis model is invoked to calculate the correlation degree of feature data within the same spatiotemporal dimension group set; a correlation degree threshold is set, feature data with a correlation degree higher than the preset correlation degree threshold is retained, and redundant feature data without correlation are filtered out; The real-time correlation analysis model is invoked to perform cross-group matching of feature data of device identifiers that have business correlations within different spatiotemporal dimension group sets, and to identify abnormal cross-device correlation features. Count the number of abnormal correlation features and set the correlation weight for each abnormal correlation feature; The spatiotemporal correlation rules of the real-time correlation analysis model include time window correlation rules and topology correlation rules; wherein, the time window correlation rules are used to define the abnormal triggering relationship between different monitoring indicators within the same preset time window; the topology correlation rules are used to define the abnormal propagation relationship based on the business topology of the IT system.

6. A real-time fusion analysis method for multi-source monitoring data according to claim 1 or 2, characterized in that, Step S5 specifically includes: Calculate the weighted total of abnormal association features; If the weighted total of abnormal correlation features exceeds the preset hierarchical alarm threshold, the IT system is judged to be operating abnormally. Features with consistent device identifiers among the abnormal correlation features are aggregated to locate the associated devices of the abnormal fault; an emergency alarm is triggered and the fault handling process is linked to send alarms to alarm terminals on multiple platforms. If the weighted total of abnormal correlation features does not exceed the graded alarm threshold, the IT system is judged to be operating normally, and the process returns to step S1. The preset hierarchical alarm thresholds for the abnormal correlation features are classified according to the importance level of the IT system.

7. A real-time fusion analysis device for multi-source monitoring data, characterized in that, include: The multi-source data acquisition module includes: Collect heterogeneous monitoring data from multiple sources in the IT system; The standardized parsing module contains: Heterogeneous monitoring data is standardized and parsed through a multi-source data access adaptation matrix to generate standard data frames in a unified format; the standard data frames are then time-stamped and calibrated. The feature extraction module contains: The calibrated standard data frame is subjected to feature extraction and filtering using a data feature extraction matrix to obtain a fused feature set. The correlation analysis module contains: Construct a real-time correlation analysis model to perform spatiotemporal correlation matching on the fused feature set and filter out irrelevant redundant feature data; Identify abnormal association features from the associated feature data, count the number of abnormal association features, and set the association weight for each abnormal association feature; The alarm output module contains: Calculate the weighted total of abnormal correlation features, and generate a graded alarm threshold based on the weighted total of abnormal correlation features; If the weighted total of abnormal correlation features exceeds the preset hierarchical alarm threshold, the IT system is judged to be operating abnormally, and the alarm information is sent to the linked multi-platform alarm terminal. If the weighted total of abnormal correlation features does not exceed the tiered alarm threshold, the IT system is judged to be operating normally, and multi-source heterogeneous data collection and analysis will continue.

8. The real-time fusion analysis device for multi-source monitoring data according to claim 7, characterized in that, The heterogeneous monitoring data in the multi-source data acquisition module specifically includes: Device metrics, log data, alarm data, and traffic data of servers, network devices, databases, and middleware in IT systems; The heterogeneous monitoring data is collected by the built-in Agent collection unit, SNMP collection unit, API collection unit, and log collection unit. The Agent collection unit is used to collect the device indicator data of the server in the IT system, the SNMP collection unit is used to collect the traffic data of the network device in the IT system, the API collection unit is used to collect the operating status data of the database and middleware in the IT system, and the log collection unit is used to collect the log data and alarm data of each device in the IT system.

9. The real-time fusion analysis device for multi-source monitoring data according to claim 7, characterized in that, The process of establishing the multi-source data access adaptation matrix in the standardized parsing module includes: Based on the format specifications of each monitoring data, the core parsing fields of each monitoring data are determined; A unified field mapping rule is set for each type of core parsing field to construct a conversion relationship from heterogeneous monitoring data to standard data frames; the conversion relationship can parse the collected heterogeneous monitoring data of various types. Construct a multi-source data access adaptation matrix based on the transformation relationship; The features in the standard data frame include: data type, data value, acquisition time, time interval, device identifier, device type, cluster to which the device belongs, indicator name, indicator threshold, indicator deviation rate, whether it is abnormal, abnormal level, and abnormal type.

10. A real-time fusion analysis device for multi-source monitoring data according to claim 7, characterized in that, The data feature extraction matrix processing procedure in the feature extraction module includes: Identify the core features required for heterogeneous monitoring data fusion analysis; filter and retain the core features from standard data frames: data type, data value, acquisition time, device identifier, device type, indicator name, whether it is abnormal, abnormal level, and abnormal type; The filtered features and their corresponding data are structured and encapsulated to obtain a fused feature set; Each feature data in the fused feature set contains a unique device identifier and a timestamp identifier.

Images